I'm a bit confused why people are freaking out about these term changes - Meta scrapes my website already on a fairly regular basis (even with robots.txt). Is the point of this change to justify more intense scraping?
still have the ability to disable the crawler via robots.txt flags
Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting first RST from web daemon and second RST from the OS.
robots.txt can used as a reward mechanism. If an automated visitor that we would normally block begins by accessing robots.txt that's a good sign and we give that a plus score or whatever. If subsequent accesses from that visitor comply with robots.txt, we let them continue (perhaps with some rate checks).
Conversely, if an obviously automated visitor does not fetch robots.txt, that can be an instant ban, kind of like a reverse honey pot, if you will.
So it can have some nonzero value in the overall context; it's just silly to think of it as a blocking mechanism.
In short, robots.txt documents what access is allowed; it states the policy. Enforcing it the job of something else.
Someone you work for in this capacity shouldn't ask about robots.txt because that's the sort of detail for which they have you to deal with.
I have been writing bots and scrapers for almost 20 years.
I've enjoyed the arms race with scrapers as they have taught me a lot over the years. Some of them leave me funny messages in my logs and seem to enjoy working around my crude mitigations almost as if it were a puzzle. Which of your scrapers can parse this post without the help of a real browser or headless chrome? [1] I am curious how far the tools have evolved and how someone with your experience instruments them if you are willing to share.
And may be or could be part of the solution by them taking upon themself to implement it in an ethical manner by having the bots and scrapers respect robots.txt; unless the spec specifically says not to in that case I agree with finding new clients or another job
This implies the solution is every actor in the world willfully complying which doesn’t sound very realistic if you ask me.
Similar to saying world hunger would be solved in everyone who could donated $100 to those in need. Sure it may be true but foot stomping about why isn’t it organically happening is missing the point.
The problem itself does not matter. If the nature of any solution relies on willful global compliance, it’s not going to be very effective at whatever it’s trying to solve.
Every society fundamentally relies on willful compliance. Sure, you can force a few outsiders, but once a sizeable population stops being compliant, it just breaks down.
I'm saying telling people they shouldn't do something is about as useless as farting in the wind. Blaming someone does nothing, it's the force of action behind the words that mean something.
But, because this person could be anyone, anywhere your words are hollow and empty. You are impotent to stop their future actions.
Yeah it reminds me of the typical copyright notice found in a book, even decades ago:
> All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.
For AI, wasn't the book first "reproduced or transmitted" into a training system?
Returning some text files from the server fs is _wayyy_ lighter than letting the bot go through the search, middleware, and db while crawling your stuff.
I'm not convinced that it's possible to poison AI. Real data all correlates, fake data is all unique in its own way. I suspect feeding AI fake data is a waste of time.
Angel banning is better, 100% agreed on giving it false data that the scraper builders can't easily detect. Give it seemingly correct and useful but totally wrong answers
I would assume it's exactly what they described, banning something by pretending to still helpfully assist them while silently undermining them instead.
You can but you'll probably want to do it with a browser extension or an undetectable webdriver.
I suggest doing it, actually. Could be a fun project. I suggest not making the script make any extra requests on your main account just in case - anything already on the page is free for scraping.
My suggestion is get your business the f*ck off Facebook.
I would never in a thousand years accept such a one-sided, despicable business relationship as their proposed terms seek to establish.
It's already bad enough they hoover up your customer lists and cross-link to their own - for purposes of gaining insight on individuals who never consented to it.
Now they're trying to use their monopoly over a form of social networking to bully you into arbitrary contract terms for a largely unrelated product.
You don't need Meta as much as you think. And in aggregate, they need their customers far more than those customers need them.
There are much friendlier ecosystems in which to engage with your users.
Regarding the customer lists, I have personal experience.
My bank advertises with Meta and included my account in some kind of upload list (probably Meta Ads Manager). I noticed their logo while poking around in Facebook's settings, under Audience-based Advertising. I was livid.
As far as I'm concerned they had no business telling a creepy company like Meta who I bank with. I certainly never consented to that.
I dug up my original banking agreement - a 5-page document on aging, yellowed parchment that I signed over a decade ago. Nothing in there about them being able to unilaterally change terms without notice. I also printed out their latest, 30+ page standard banking agreement which (after making inquiries) they seemed to think applied to my account.
I brought both in to my bank manager and put them on the table. At a high level, I pointed how ridiculous the size had grown to, and he was sympathetic.
In terms of details, I'd highlighted several portions that pertain to privacy, and we walked through the differences together. I underscored how none of them were needed to provide me banking services, how the benefits they conveyed were solely to TD, and how they were harmful to me as a customer.
In the end, he agreed to formally acknowledge on file that I did not agree to the new terms.
You may disagree over how much effect that will have in practice, but the acknowledgment was not just a bureaucratic exercise. It established very firm evidence that I never agreed to the new contract, or any terms the bank might rely on as a basis for the objectionable practices. It would become material if those practices cause me damage in the future, significant enough to litigate. i.e. They knew they lacked agreement, and continued to provide me banking services.
He also worked with internal departments to toggle all available opt-out flags.
I was fortunate in this case the manager comprehended my concerns and seemed to want to keep me as a customer. I recognize not everyone would necessarily get as far.
It really disgusts me that some of the most degenerate practices of silicon valley are spreading to other industries. With lawyers more broadly adopting some of the most viciously one-sided legal language big tech pioneered.
I for one am not on board, and refuse to be bullied into providing some kind of imaginary consent. I say 'imaginary', because I don't think anyone in their right mind who reads and understands such terms would ever agree to them willingly.
[resource]+[mountain/stone/field/valley/tree] in German is a very common pattern for Jewish surnames. A quirk of how surnames were adopted in German-speaking regions in the 18th century
Sugar mountain at least makes sense, unlike e.g. wine stone
I read somewhere once that Sam Altman sounds like a name for a fictional villain made up by Hideo Kojima for Metal Gear Solid, and it will never not make sense.
Color me shocked... one of the most profitable companies in the world, that doesn't contribute anything positive to society, is unethical. Just another day in America that ends in Y.
Blocking by UA is pointless now, meta-externalagent rotates and a chunk comes off non-Meta ASNs. We matched on ASN plus TLS JA4 and that actually stuck.
Your robots.txt "Do Not Scrape" signs become legally useless: robots.txt is just a voluntary preference, not a law. While Meta says in PR statements that it will respect robots.txt, it left that promise out of the binding legal contract. By keeping your business account active, you are signing a contract that legally authorizes Meta to ignore your site’s block-lists.
- Meta can create fake accounts to bypass registration walls: The terms explicitly authorize Meta to "create test credentials" to crawl your site. Legal experts warn this gives Meta permission to sign up as a fake user to scrape gated content behind login screens or basic paywalls that anonymous bots usually can't see.
- Your entire domain is fair game, not just links you post: Because the terms cover any domain "associated with your products or services" (or shared by regular users on Meta), Meta isn't just scraping the specific landing pages you run ads for—it has a green light to crawl your entire website from top to bottom.
- AI training is still allowed through a backdoor: Removing the word "train" is a PR smokescreen. The contract still lets Meta scrape your site to "develop, improve, and research... advertising and business products." Because "business products" isn't strictly defined—and Meta builds AI tools for businesses "developing" legally covers training AI models on your proprietary articles, data, and work.
- Third-party companies get the same access: The terms grant these scraping rights to "any entity acting on our behalf," meaning external contractors or third-party scraping vendors hired by Meta also have legal cover to harvest your site.
- It is a non-negotiable hostage situation (Zero Opt-Out): There is no toggle or form to reject this before the October 30 deadline. Your only legal way to decline is to completely delete your commercial presence across Facebook, Instagram, Threads, and WhatsApp—forcing you to choose between surrendering your content for free or losing access to your audience.
The Bottom Line: Starting October 30, their updated terms give it sweeping permission to scrape entire websites linked to any Meta Business account (Facebook, Instagram, Threads, WhatsApp). Despite a last-minute wording tweak after backlash, legal experts warn the rules still let Meta harvest publisher content to build AI tools—with zero option to opt out.
1. What Exact Things Can Meta Do?
If you have a Meta Business account, you authorize Meta (and its third-party partners) to:
a) Deploy automated bots: Use web scrapers, crawlers, spiders, and user agents on your site.
b) Bypass basic walls: Create "test credentials" to access, index, cache, analyze, and collect your content.
c) Monitor continuously: Periodically pull updates from your domains, web pages, and URLs.
2. What is the Extent of the Scraping?
- What gets scraped: Any domain, webpage, URL, or content "associated with" your business or shared by users on Meta platforms. Legal experts note this is broad enough to let Meta scrape entire websites, not just specific landing pages.
-
- Who it affects: Virtually every news organization and business that uses a Meta account to manage Facebook Pages, Instagram profiles, Threads, or WhatsApp channels.
How it expanded:
- Previously: Meta could only scrape your site if you specifically used its generative AI ad features, and only to "optimize your ads."
- Now: It applies to all business accounts by default, and scraped data can be used for all "advertising and business products" (plus platform security/integrity).
3. Why Legal Experts Say Meta's "Fix" Changes Nothing
After initial outrage, Meta revised the text on Friday to remove the word "train" and claimed it still respects robots.txt (the code websites use to block scrapers). Legal experts aren't buying it for three reasons:
- "Develop" still means "Train": Meta replaced "train" with permission to "develop, improve, research, and maintain advertising and business products." Attorneys point out that "business products" isn't strictly defined, and "developing" AI business tools naturally includes training AI models on your data.
- The robots.txt loophole: While a Meta spokesperson publicly promised to respect robots.txt flags, Meta refused to put that promise into the actual legal terms. Because robots.txt is voluntary, signing a contract that explicitly authorizes scraping legally overrides your robots.txt preferences.
- Take it or leave it: There is no opt-out. To avoid giving Meta permission to scrape their websites, businesses and news outlets would have to abandon Facebook, Instagram, Threads, and WhatsApp entirely before October 30.
If that were true, you wouldn’t have people shouting into the wind demanding that billions of people stop using Instagram, WhatsApp, Muse, Facebook, Quest, etc.
Other people don’t share your values. You’ll have to live with it.
I'm a bit confused why people are freaking out about these term changes - Meta scrapes my website already on a fairly regular basis (even with robots.txt). Is the point of this change to justify more intense scraping?
I'm with you. I don't quite grasp the magnitude of response.
still have the ability to disable the crawler via robots.txt flags
Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting first RST from web daemon and second RST from the OS.
I have been writing bots and scrapers for almost 20 years.
Not once have I had someone I work for in this capacity even ask about robots.txt
How much would I have to pay you for you to change careers permanently?
It wouldn't make much difference. Me, I just do it for fun.
robots.txt can used as a reward mechanism. If an automated visitor that we would normally block begins by accessing robots.txt that's a good sign and we give that a plus score or whatever. If subsequent accesses from that visitor comply with robots.txt, we let them continue (perhaps with some rate checks).
Conversely, if an obviously automated visitor does not fetch robots.txt, that can be an instant ban, kind of like a reverse honey pot, if you will.
So it can have some nonzero value in the overall context; it's just silly to think of it as a blocking mechanism.
In short, robots.txt documents what access is allowed; it states the policy. Enforcing it the job of something else.
Someone you work for in this capacity shouldn't ask about robots.txt because that's the sort of detail for which they have you to deal with.
I have been writing bots and scrapers for almost 20 years.
I've enjoyed the arms race with scrapers as they have taught me a lot over the years. Some of them leave me funny messages in my logs and seem to enjoy working around my crude mitigations almost as if it were a puzzle. Which of your scrapers can parse this post without the help of a real browser or headless chrome? [1] I am curious how far the tools have evolved and how someone with your experience instruments them if you are willing to share.
[1] - https://altchan.org/ot/20
You are part of the problem, sounds like.
> You are part of the problem, sounds like.
And may be or could be part of the solution by them taking upon themself to implement it in an ethical manner by having the bots and scrapers respect robots.txt; unless the spec specifically says not to in that case I agree with finding new clients or another job
This implies the solution is every actor in the world willfully complying which doesn’t sound very realistic if you ask me.
Similar to saying world hunger would be solved in everyone who could donated $100 to those in need. Sure it may be true but foot stomping about why isn’t it organically happening is missing the point.
There is a clear difference between the global issue of famine and actively choosing to work for an arsehole corp.
The problem itself does not matter. If the nature of any solution relies on willful global compliance, it’s not going to be very effective at whatever it’s trying to solve.
Every society fundamentally relies on willful compliance. Sure, you can force a few outsiders, but once a sizeable population stops being compliant, it just breaks down.
Hence why the internet doesn't work well.
The internet is not a society. The internet is all societies. Hell, more and more it's not even human societies.
Denied. One can choose not to be an active element in that problem.
Yay, you solved murder, give yourself a good pat on the back.
You're saying that murderers shouldn't be blamed for murdering?
I'm saying telling people they shouldn't do something is about as useless as farting in the wind. Blaming someone does nothing, it's the force of action behind the words that mean something.
But, because this person could be anyone, anywhere your words are hollow and empty. You are impotent to stop their future actions.
So, what’s your proposed solution?
Yeah, calling this "disabling" it is inaccurate, it's just "asking nicely".
Yeah it reminds me of the typical copyright notice found in a book, even decades ago:
> All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.
For AI, wasn't the book first "reproduced or transmitted" into a training system?
How would we reformulate it today?
Legal notices are designed to be as scary as possible so they usually say you can't do things that you actually can do.
See also: Nintendo
Wouldn't feeding them false data be even better?
That could consume as much, if not more, resources as returning real data
Returning some text files from the server fs is _wayyy_ lighter than letting the bot go through the search, middleware, and db while crawling your stuff.
Or just sending it a tiny zip bomb, or even just some sh*t right out of /dev/random
/dev/random is much too expensive for this.
flat files, a tarpit, maybe zip bombs as the sibling suggested. Nothing calculated. I'd go with zipped Brothers Grimm
Bots are pretty good at detecting and preventing this kind of issue. Dumb clients would still be affected.
We've already been using AI to target military strikes without human intervention.
Make of that what you will.
I'm not convinced that it's possible to poison AI. Real data all correlates, fake data is all unique in its own way. I suspect feeding AI fake data is a waste of time.
I'd say it depends what exactly. If it's a rare topic then you'll have more luck.
Angel banning is better, 100% agreed on giving it false data that the scraper builders can't easily detect. Give it seemingly correct and useful but totally wrong answers
Would you mind explaining what "angel banning" is? I've never heard this term before and even Google doesn't seem to know what it is.
I would assume it's exactly what they described, banning something by pretending to still helpfully assist them while silently undermining them instead.
The accepted term is actually heaven banning.
And they spend huge resource so that we can't scrap Facebook - our own feed.
You can but you'll probably want to do it with a browser extension or an undetectable webdriver.
I suggest doing it, actually. Could be a fun project. I suggest not making the script make any extra requests on your main account just in case - anything already on the page is free for scraping.
> so that we can't scrap Facebook
Whilst I assume you meant to say “scrape”, I can’t help but appreciate the poetry of that typo.
My suggestion is get your business the f*ck off Facebook.
I would never in a thousand years accept such a one-sided, despicable business relationship as their proposed terms seek to establish.
It's already bad enough they hoover up your customer lists and cross-link to their own - for purposes of gaining insight on individuals who never consented to it.
Now they're trying to use their monopoly over a form of social networking to bully you into arbitrary contract terms for a largely unrelated product.
You don't need Meta as much as you think. And in aggregate, they need their customers far more than those customers need them.
There are much friendlier ecosystems in which to engage with your users.
Regarding the customer lists, I have personal experience.
My bank advertises with Meta and included my account in some kind of upload list (probably Meta Ads Manager). I noticed their logo while poking around in Facebook's settings, under Audience-based Advertising. I was livid.
As far as I'm concerned they had no business telling a creepy company like Meta who I bank with. I certainly never consented to that.
I dug up my original banking agreement - a 5-page document on aging, yellowed parchment that I signed over a decade ago. Nothing in there about them being able to unilaterally change terms without notice. I also printed out their latest, 30+ page standard banking agreement which (after making inquiries) they seemed to think applied to my account.
I brought both in to my bank manager and put them on the table. At a high level, I pointed how ridiculous the size had grown to, and he was sympathetic.
In terms of details, I'd highlighted several portions that pertain to privacy, and we walked through the differences together. I underscored how none of them were needed to provide me banking services, how the benefits they conveyed were solely to TD, and how they were harmful to me as a customer.
In the end, he agreed to formally acknowledge on file that I did not agree to the new terms.
You may disagree over how much effect that will have in practice, but the acknowledgment was not just a bureaucratic exercise. It established very firm evidence that I never agreed to the new contract, or any terms the bank might rely on as a basis for the objectionable practices. It would become material if those practices cause me damage in the future, significant enough to litigate. i.e. They knew they lacked agreement, and continued to provide me banking services.
He also worked with internal departments to toggle all available opt-out flags.
I was fortunate in this case the manager comprehended my concerns and seemed to want to keep me as a customer. I recognize not everyone would necessarily get as far.
It really disgusts me that some of the most degenerate practices of silicon valley are spreading to other industries. With lawyers more broadly adopting some of the most viciously one-sided legal language big tech pioneered.
I for one am not on board, and refuse to be bullied into providing some kind of imaginary consent. I say 'imaginary', because I don't think anyone in their right mind who reads and understands such terms would ever agree to them willingly.
just rename the company to Meta Law or Zuckerberg & Associates already
Meta law actually sounds pretty good to me.
[flagged]
[resource]+[mountain/stone/field/valley/tree] in German is a very common pattern for Jewish surnames. A quirk of how surnames were adopted in German-speaking regions in the 18th century
Sugar mountain at least makes sense, unlike e.g. wine stone
> wine stone
https://en.wikipedia.org/wiki/Potassium_bitartrate
Not strange. It's a Jewish name.
What about Altman?
I read somewhere once that Sam Altman sounds like a name for a fictional villain made up by Hideo Kojima for Metal Gear Solid, and it will never not make sense.
Saltman
Does it really matter what they have "permission" for when they'll do it anyway?
Glad I never signed up on any of their "services", and never accepted any of their terms.
The valuable data to scrape is in the IDE of the expert and that expert getting challenges.
Color me shocked... one of the most profitable companies in the world, that doesn't contribute anything positive to society, is unethical. Just another day in America that ends in Y.
Blocking by UA is pointless now, meta-externalagent rotates and a chunk comes off non-Meta ASNs. We matched on ASN plus TLS JA4 and that actually stuck.
[flagged]
[dead]
TLDR;
The Danger List
Your robots.txt "Do Not Scrape" signs become legally useless: robots.txt is just a voluntary preference, not a law. While Meta says in PR statements that it will respect robots.txt, it left that promise out of the binding legal contract. By keeping your business account active, you are signing a contract that legally authorizes Meta to ignore your site’s block-lists.
- Meta can create fake accounts to bypass registration walls: The terms explicitly authorize Meta to "create test credentials" to crawl your site. Legal experts warn this gives Meta permission to sign up as a fake user to scrape gated content behind login screens or basic paywalls that anonymous bots usually can't see.
- Your entire domain is fair game, not just links you post: Because the terms cover any domain "associated with your products or services" (or shared by regular users on Meta), Meta isn't just scraping the specific landing pages you run ads for—it has a green light to crawl your entire website from top to bottom.
- AI training is still allowed through a backdoor: Removing the word "train" is a PR smokescreen. The contract still lets Meta scrape your site to "develop, improve, and research... advertising and business products." Because "business products" isn't strictly defined—and Meta builds AI tools for businesses "developing" legally covers training AI models on your proprietary articles, data, and work.
- Third-party companies get the same access: The terms grant these scraping rights to "any entity acting on our behalf," meaning external contractors or third-party scraping vendors hired by Meta also have legal cover to harvest your site.
- It is a non-negotiable hostage situation (Zero Opt-Out): There is no toggle or form to reject this before the October 30 deadline. Your only legal way to decline is to completely delete your commercial presence across Facebook, Instagram, Threads, and WhatsApp—forcing you to choose between surrendering your content for free or losing access to your audience.
The Bottom Line: Starting October 30, their updated terms give it sweeping permission to scrape entire websites linked to any Meta Business account (Facebook, Instagram, Threads, WhatsApp). Despite a last-minute wording tweak after backlash, legal experts warn the rules still let Meta harvest publisher content to build AI tools—with zero option to opt out.
1. What Exact Things Can Meta Do?
If you have a Meta Business account, you authorize Meta (and its third-party partners) to:
a) Deploy automated bots: Use web scrapers, crawlers, spiders, and user agents on your site. b) Bypass basic walls: Create "test credentials" to access, index, cache, analyze, and collect your content. c) Monitor continuously: Periodically pull updates from your domains, web pages, and URLs.
2. What is the Extent of the Scraping? - What gets scraped: Any domain, webpage, URL, or content "associated with" your business or shared by users on Meta platforms. Legal experts note this is broad enough to let Meta scrape entire websites, not just specific landing pages. - - Who it affects: Virtually every news organization and business that uses a Meta account to manage Facebook Pages, Instagram profiles, Threads, or WhatsApp channels.
How it expanded:
3. Why Legal Experts Say Meta's "Fix" Changes Nothing After initial outrage, Meta revised the text on Friday to remove the word "train" and claimed it still respects robots.txt (the code websites use to block scrapers). Legal experts aren't buying it for three reasons:- "Develop" still means "Train": Meta replaced "train" with permission to "develop, improve, research, and maintain advertising and business products." Attorneys point out that "business products" isn't strictly defined, and "developing" AI business tools naturally includes training AI models on your data.
- The robots.txt loophole: While a Meta spokesperson publicly promised to respect robots.txt flags, Meta refused to put that promise into the actual legal terms. Because robots.txt is voluntary, signing a contract that explicitly authorizes scraping legally overrides your robots.txt preferences.
- Take it or leave it: There is no opt-out. To avoid giving Meta permission to scrape their websites, businesses and news outlets would have to abandon Facebook, Instagram, Threads, and WhatsApp entirely before October 30.
Can't you place your meta accounts under a separate legal entity?
Zuckerberg is an absolute sociopath, a cancer on the internet, he should've been shut down years ago
[flagged]
[flagged]
[flagged]
I for sure don't love any of the oligarchs, but how is Zuck worse than e.g. Musk?
Similar resources and anti-social intent, but better organized and disciplined.
Why not both? True though, after Musk's Doge stunt he has caused a lot of real harm rivalling that of Zuck.
Musk may be worse, but at least he makes cars and sells Ubereats for space. Zuck produces nothing of value.
If that were true, you wouldn’t have people shouting into the wind demanding that billions of people stop using Instagram, WhatsApp, Muse, Facebook, Quest, etc.
Other people don’t share your values. You’ll have to live with it.