Technology

59582 readers

2667 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

344

OpenAI and Anthropic are ignoring an established rule that prevents bots scraping online content (archive.ph)

submitted 5 months ago by IndustryStandard@lemmy.world to c/technology@lemmy.world

55 comments fedilink hide all child comments

The world's top two AI startups are ignoring requests by media publishers to stop scraping their web content for free model training data, Business Insider has learned.

OpenAI and Anthropic have been found to be either ignoring or circumventing an established web rule, called robots.txt, that prevents automated scraping of websites.

TollBit, a startup aiming to broker paid licensing deals between publishers and AI companies, found several AI companies are acting in this way and informed certain large publishers in a Friday letter, which was reported earlier by Reuters. The letter did not include the names of any of the AI companies accused of skirting the rule.

OpenAI and Anthropic have stated publicly that they respect robots.txt and blocks to their specific web crawlers, GPTBot and ClaudeBot.

However, according to TollBit's findings, such blocks are not being respected, as claimed. AI companies, including OpenAI and Anthropic, are simply choosing to "bypass" robots.txt in order to retrieve or scrape all of the content from a given website or page.

A spokeswoman for OpenAI declined to comment beyond pointing BI to a corporate blogpost from May, in which the company says it takes web crawler permissions "into account each time we train a new model." A spokesperson for Anthropic did not respond to emails seeking comment.

Robots.txt is a single bit of code that's been used since the late 1990s as a way for websites to tell bot crawlers they don't want their data scraped and collected. It was widely accepted as one of the unofficial rules supporting the web.

you are viewing a single comment's thread
view the rest of the comments

[–] PythagreousTitties@lemm.ee 148 points 5 months ago (1 children)

Oh boy, if they're ignoring robots txt, then I better ...add a useless link at the bottom of every comment I make. That'll really show them!

[–] cbarrick@lemmy.world 72 points 5 months ago* (last edited 5 months ago) (2 children)

This comment is copyrighted by me and licensed to the public under the terms of the CC-BY-NC-SA 4.0. If you intend to use this comment for commercial purposes, you must secure a commercial license from me, which will cost you a lot of money. If you violate the terms of the CC-BY-NC-SA 4.0 without securing an appropriate license, I will send my army of lawyers that I totally definitely have to defend my copyright against you in court.

[–] SturgiesYrFase@lemmy.ml 40 points 5 months ago (3 children)

Exactly, I never understood what people thought they would achieve by putting the link to that in their comments. Like, AI firms are absolutely willing to skim through copyrighted works of artists, backed by a much stronger license, what makes you think linking that will achieve anything. Except maybe poisoning the LLM well.
Hey, there's a thought. If we all just put that at the end of every comment, I wonder if GPT6 will figure that's just how people talk and end all it's responses with it?

[–] rebelsimile@sh.itjust.works 18 points 5 months ago

All they’re going to do is teach the AI that sometimes people end posts with useless disclaimers.

[–] CheeseNoodle@lemmy.world 9 points 5 months ago (2 children)

I've been buffalo buffalo buffalo buffalo buffalo trying to remember to put some nonsense somewhere in my comments every time in order to make the LLMs think this is how people talk.

[–] Hossenfeffer@feddit.uk 6 points 5 months ago

I'm anaspeptic, phrasmotic, even compunctuous to have read such pericombobulation.

[–] SturgiesYrFase@lemmy.ml 6 points 5 months ago

That's quite quiet quintessentially quiescent and clearly colloquially colour clever.

[–] Zoboomafoo 6 points 5 months ago (1 children)

They figure no one person will have the means to sue or the ability to prove that their data was scraped.

[–] Corkyskog@sh.itjust.works 6 points 5 months ago

Could we take em to small claims for like $500 a comment? That would be a devastating movement lol

[–] match@pawb.social 17 points 5 months ago

This media provided under the CC-Mine-Now license. You may remix this comment for use in your AI feeding but if you do I own your company now and all proceeds go to me, and you indemnify me against anything I want to do to your company.