The tech giant is sidestepping guardrails that websites use to prevent being scraped, data show, in a move whistleblowers say is unethical and potentially illegal.
Meta has scraped data from the most-trafficked domains on the internet —including news organizations, education platforms, niche forums, personal blogs, and even revenge porn sites—to train its artificial intelligence models, according to a leaked list obtained by Drop Site News.
By scraping data from roughly 6 million unique websites, including 100,000 of the top-ranked domains, Meta has generated millions of pages of content to use for Meta’s AI-training pipeline.
The sites that Meta scrapes consist of copyrighted content, pirated content, and adult videos, some of whose content is potentially illegally obtained or recorded, as well as news and original content from prominent outlets and content publishers.
They include mainstream businesses like Getty Images, Shopify, Shutterstock, but also extreme pornographic content, including websites advertising explicit sexual content and humiliation porn that exploits teenagers.
IP debate aside, LLM scrapers absolutely annihilate system resources. I host a wordpress site and before setting up cloudflare labyrinth my whole server would get ddos’d at least twice a day.
I blocked the entire ASN for Meta, because they were downright dirty with their scraping. No gradual crawling, fakes UAs, random addresses across a large number of subnets.
They weren’t the only ones either. The AI scraping heist is the new goldrush.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: [email protected]
This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.
Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.
Rules:
1: All Lemmy rules apply
2: Do not post low effort posts
3: NEVER post naziped*gore stuff
4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.
5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)
6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist
7: crypto related posts, unless essential, are disallowed
My Mastodon instance is on the list. I try hard to block them.
The problem with the list is that it’s a target list, but not a list showing how much content, if any, they manage to process from any of the sites.
Stole.
We ban the petty scrapers and celebrate the great ones as innovators and promote them as fortune 500 companies.
Good catch. That’s worth a seperate post.
Hexbear is on the list too.
Llms will start randomly shitting out hexbear emotes lol
where did you get the .txt file?
Just extracted it from the PDF
Always amused when leftist instances treat intellectual property like it’s real.
IP debate aside, LLM scrapers absolutely annihilate system resources. I host a wordpress site and before setting up cloudflare labyrinth my whole server would get ddos’d at least twice a day.
its not, but scraping is annoyingly resource intensive.
One person’s “scraping” is another person’s plagiarism.
I blocked the entire ASN for Meta, because they were downright dirty with their scraping. No gradual crawling, fakes UAs, random addresses across a large number of subnets.
They weren’t the only ones either. The AI scraping heist is the new goldrush.