The rise and fall of robots.txt
www.theverge.com
external-link
As unscrupulous AI companies crawl for more and more data, the basic social contract of the web is falling apart.
@Dkarma@lemmy.world
link
fedilink
English
-338M

removed by mod

@rtxn@lemmy.world
link
fedilink
English
43
edit-2
8M

I would be shocked if any big corpo actually gave a shit about it, AI or no AI.

if exists("/robots.txt"):
    no it fucking doesn't
@DingoBilly@lemmy.world
link
fedilink
English
18M

Yeah I always found it surprising that everyone just agreed to follow a text file on a website on how to act. It’s one of the worst thought out/significant issues with browsing still out there from the beginning pretty much.

@spez_@lemmy.world
link
fedilink
English
-138M

Why should I care about a text file lol

@archchan@lemmy.ml
link
fedilink
English
1
edit-2
8M

deleted by creator

Cosmic Cleric
link
fedilink
English
848M

As unscrupulous AI companies crawl for more and more data, the basic social contract of the web is falling apart.

Honestly it seems like in all aspects of society the social contract is being ignored these days, that’s why things seem so much worse now.

@maness300@lemmy.world
link
fedilink
English
128M

It’s abuse, plain and simple.

kingthrillgore
link
fedilink
English
168M

I explicitly have my robots.txt set to block out AI crawlers, but I don’t know if anyone else will observe the protocol. They should have tools I can submit a sitemap.xml against to know if i’ve been parsed. Until they bother to address this, I can only assume their intent is hostile and if anyone is serious about building a honeypot and exposing the tooling for us to deploy at large, my options are limited.

Optional
link
fedilink
English
668M

Well the trump era has shown that ignoring social contracts and straight up crime are only met with profit and slavish devotion from a huge community of dipshits. So. Y’know.

@FrankTheHealer@lemmy.world
link
fedilink
English
15
edit-2
5M

deleted by creator

what is it?

@mrnarwall@lemmy.world
link
fedilink
English
128M

Robots.txt is a file that is is accessible as part of an http request. It’s a backend configuration file that sets rules for what automatically running web crawlers are allowed. It can set both who is and who isn’t allowed. Google is usually the most widely allowed domain for bots just because their crawler is how they find websites for search results. But it’s basically the honor system. You could write a scraper today that goes to websites that it is being told it doesn’t have permission to view this page, ignore it, and still get the information

@Ascend910@lemmy.ml
link
fedilink
English
88M

This is a very interesting read. It is very rarely people on the internet agree to follow 1 thing without being forced

@masonlee@lemmy.world
link
fedilink
English
-48M

Also, by the way, violating a basic social contract to not work towards triggering an intelligence explosion that will likely replace all biological life on Earth with computronium, but who’s counting? :)

@lunarul@lemmy.world
link
fedilink
English
28M

That would be a danger if real AI existed. We are very far away from that and what is being called “AI” today (which is advanced ML) is not the path to actual AI. So don’t worry, we’re not heading for the singularity.

@masonlee@lemmy.world
link
fedilink
English
08M

I request sources :)

@lunarul@lemmy.world
link
fedilink
English
18M

https://www.lifewire.com/strong-ai-vs-weak-ai-7508012

Strong AI, also called artificial general intelligence (AGI), possesses the full range of human capabilities, including talking, reasoning, and emoting. So far, strong AI examples exist in sci-fi movies

Weak AI is easily identified by its limitations, but strong AI remains theoretical since it should have few (if any) limitations.

https://en.m.wikipedia.org/wiki/Artificial_general_intelligence

As of 2023, complete forms of AGI remain speculative.

Boucher, Philip (March 2019). How artificial intelligence works

Today’s AI is powerful and useful, but remains far from speculated AGI or ASI.

https://www.itu.int/en/journal/001/Documents/itu2018-9.pdf

AGI represents a level of power that remains firmly in the realm of speculative fiction as on date

@masonlee@lemmy.world
link
fedilink
English
18M

Ah, I understand you now. You don’t believe we’re close to AGI. I don’t know what to tell you. We’re moving at an incredible clip; AGI is the stated goal of the big AI players. Many experts think we are probably just one or two breakthroughs away. You’ve seen the surveys on timelines? Years to decades. Seems wise to think ahead to its implications rather than dismiss its possibility.

@lunarul@lemmy.world
link
fedilink
English
18M

See the sources above and many more. We don’t need one or two breakthroughs, we need a complete paradigm shift. We don’t even know where to start with for AGI. There’s a bunch of research, but nothing really came out of it yet. Weak AI has made impressive bounds in the past few years, but the only connection between weak and strong AI is the name. Weak AI will not become strong AI as it continues to evolve. The two are completely separate avenues of research. Weak AI is still advanced algorithms. You can’t get AGI with just code. We’ll need a completely new type of hardware for it.

@masonlee@lemmy.world
link
fedilink
English
18M

Before Deep Learning recently shifted the AI computing paradigm, I would have written exactly what you wrote. But as of late, the opinion that we need yet another type of hardware to surpass human intelligence seems increasingly rare. Multimodal generative AI is already pretty general. To count as AGI for you, you would like to see the addition of continuous learning and agentification? (Or are you looking for “consciousness”?)

That said, I’m all for a new paradigm, and favor Russell’s “provably beneficial AI” approach!

@lunarul@lemmy.world
link
fedilink
English
18M

Deep learning did not shift any paradigm. It’s just more advanced programming. But gen AI is not intelligence. It’s just really well trained ML. ChatGPT can generate text that looks true and relevant. And that’s its goal. It doesn’t have to be true or relevant, it just has to look convincing. And it does. But there’s no form of intelligence at play there. It’s just advanced ML models taking an input and guessing the most likely output.

Here’s another interesting article about this debate: https://ourworldindata.org/ai-timelines

What we have today does not exhibit even the faintest signs of actual intelligence. Gen AI models don’t actually understand the output they are providing, that’s why they so often produce self-contradictory results. And the algorithms will continue to be fine-tuned to produce fewer such mistakes, but that won’t change the core of what gen AI really is. You can’t teach ChatGPT how to play chess or a new language or music. The same model can be trained to do one of those tasks instead of chatting, but that’s not how intelligence works.

I don’t think glorified predictive text is posing any real danger to all life on Earth.

@MataVatnik@lemmy.world
link
fedilink
English
28M

Until we weave consciousness with machines we should be good.

𝐘Ⓞz҉
link
fedilink
English
128M

No laws to govern so they can do anything they want. Blame boomer politicians not the companies.

@Sorgan71@lemmy.world
link
fedilink
English
-48M

good. robots.txt was always a bad idea

Like so many terrible ideas, it worked flawlessly for generations

circuitfarmer
link
fedilink
English
458M

Most every other social contract has been violated already. If they don’t ignore robots.txt, what is left to violate?? Hmm??

@olympicyes@lemmy.world
link
fedilink
English
38M

removed by mod

BlanketsWithSmallpox
link
fedilink
English
288M

It’s almost as if leaving things to social contracts vs regulating them is bad for the layperson… 🤔

Nah fuck it. The market will regulate itself! Tax is theft and I don’t want that raise or I’ll get in a higher tax bracket and make less!

And then the companies hit the “trust thermocline”, customers leave them in droves and companies wonder how this could’ve happened.

@Jimmyeatsausage@lemmy.world
link
fedilink
English
9
edit-2
8M

This can actually be an issue for poor people, not because of tax brackets but because of income-based assistance cutoffs. If $1/hr raise throws you above those cutoffs, that extra $160 could cost you $500 in food assistance, $5-$10/day for school lunch, or get you kicked out of government subsidied housing.

Yet another form of persecution that the poor actually suffer and the rich pretend to.

Create a post

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


  • 1 user online
  • 186 users / day
  • 583 users / week
  • 1.37K users / month
  • 4.49K users / 6 months
  • 1 subscriber
  • 7.41K Posts
  • 84.7K Comments
  • Modlog