Man, this sucks. I doubt there’s anything that I can say to get people to stop doing things like this, but the eventual outcome here isn’t going to be freedom for you to scrape sites that are trying to avoid being DDoSed by bots, but instead that we all end up in a world where device attestation is required to do practically anything online. And for what?
Is it ethical to scrape when a site has explicitly blocked bots? I know a fair number of people who run small sites who are already considering closing them down because the bots are relentlessly hammering their sites and driving up hosting costs.
Use cases range from sending 100s of requests per second just to bring a website down to doing a montly request to a municipality's endpoint to get a local dashboard of when trash is picked up. I don't think you can pass a single judgement of automating web requests in general.
I know for a fact companies in China do this. One paid me directly for work in this space, and in another, I found my own OSS project
in their artifact cache while working there.
I think it really depends on how the bot is run. If the bot is replacing me navigating there manually, absolutely. If it’s sucking up content to rip off and make someone else billions of dollars, no.
There actually is a well defined demarcation point where the site owners wishes do apply. It's called... A demarc point! Everything that happens past this point is by definition entirely in the control of the admin. Which includes closing the door on whoever they wish. This is a technical reality, not something you can effect with opinions or wanting things more than other people want other things. It might seem contradictory but this is actually the main thing that makes the internet as free as it is.
When we're talking about ethics, I think we should give the owner more than that. If they don't want to be DDoSed we shouldn't say "too bad, the attackers are outside your network, your wishes don't apply, get good".
If it's just impersonating me to help me better consume the content as if I were the one driving, then it's perfectly ethical, and not even related to a bot ban. Shades of gray and all...
I think that instead of trying to prevent web scraping, websites should try to make it easier so that it generates less traffic. As long as any user is allowed to view the website, there will always be a way to scrape it anyway. If there were simply a monthly updated torrent available on a standardized subpage, such as example.com/scrape, scraping would be much less harmful.
> Bot detectors flag automation by reading the browser fingerprint; Fortress corrects that fingerprint inside Chromium's C++, so the browser presents as an ordinary Chrome install.
This does not seem like it would work against anything but the most basic bot protection.
For agent/browser automation, getting blocked is only one part of the problem. The other hard part is knowing whether the page you got back is the real page, a degraded version, or some silent challenge page.
Since my grade-school mascot/team was "The Trojans" and my adopted hometown is firmly rooted in pre-Christian Greek culture and mythology, I am currently studying the Trojan War, and I may observe that "Trojan" is an extremely perspicacious brand-name for condoms, more than the average guy would expect.
I think more like mimicking the perfect resident. Then again it’s Theseus’s ship no, if you are able to do something so perfectly what is the difference between you and the clone? A Hollywood movie in the making.
Reminds me of this old joke: No ethically trained software engineer would ever write a destroyBaghdad function - they’d write a destroyCity function to which you could pass Baghdad as a parameter.
This is really unacceptable folks. There are those of us that have to keep these sites up, and it's seriously been a few years of nightmare scrapers and botnets, and stupid things like this that you are trying to legitimize that will make this worse. If a site doesn't want you, you should go away. There's a reason for it. Not every website is backed by a billion/trillion dollar company with the resources to absorb things.
Making a network request from a script is not abuse, consuming excess bandwidth is. Scrapers already spoof browser reputation and cycle IPs while abusing bandwidth. Recently I wanted to convert the results of an Autotrader filter into a table so I could supplement it with data they don’t track (towing capacity). It was two pages of results, but requesting it from a script was aggressively blocked by browser fingerprinting. I had to port it to JS and run it in my browser console manually to get the data out, wasting my time.
I think this is the point that has become a moral dilemma. Increasingly, in this agentic age tasks like research and web exploration that would be done by humans and even buy things from an website is now done through agents. While I sympathize with the main author about Ddos, but blocking every agent regardless of intent seems cruel. This would allow an MCP doing deep research(almost identical to human behavior) to move forward. But there should definitely be more work done system that focus on agent intent(i.e some agent trying to find the best price for vacation or scraper building alternatives to Travel Advisor).
Ideally, there would be a better way for web hosts to block bandwidth-hogging, but not block bots. I don't think anyone cares if some automated user-agent requests a 1KB file from their web site. They care that automated systems are sucking down TBs per day, all day, every day.
Making people actually read content doesn't strike me as cruel. Tedious, sure, but not cruel. What is cruel is seeing all of our collective individuality and artistic expression jammed into an LLM, sold back to us by the token, and forced into our lives by an economy increasingly skewed towards holders of capital.
Fair point. Again, I might be terribly and almost Icarus-like optimistic, but I still believe the push and pull between Big Tech and Indie AI builders would still lead to an equilibrium that makes the world a little more efficient and perhaps in time, more creative than AI slop. Let's hope Open Source AI opens instead of OpenAI so the internet is not feeding the capitalist machine of OpenAI and Anthropic.
CDN security SWE here. I have functionally zero interest in what individual people do in terms of automation. I automated my gym's class signup forum during COVID so I get the validity of the use case. You want to work around the mitigations to do the same? Go with my blessings.
The killer is that everything that works for individuals trying to get through the day and make the web a bit smoother is immediately used by industrial crawlers strip mining the Internet, and you can't block one without blocking the other. A future where the web is only accessible via device attestation is extremely dystopian but so is an Internet where every drop of content goes into an LLM training set.
Things like this is why all of the worthwhile content is going to drain into balkanized spaces, and we're all the poorer for it.
Hmm, I am sure you know more than I do on the topic. At least to me, an amateur in the security space, would the action of the scraper not be closer to repetitive calls to tables instead of, i.e a more inquisitive agent doing research or booking who spends more time looking through, and surely that can be tracked? Again, on the moral side, I agree that if one comes with the other, it is quite dystopian. I myself am an ML Inference Engineer, so I guess interesting problems and interesting solutions always draw me in much like this.
Abusive crawlers use residential proxies to distribute crawl traffic between thousands of IPs, so that one agent that's making five requests blends in with the massive crawl. There is signal but there are ML inference engineers tuning their traffic to blend in, which requires that mitigations must get more sensitive.
At the end of the day, the defensive side has an obligation to protect customers and well behaved agents are the first thing that gets swept up along with the bad actors.
It’s not that dystopian if you are given the choice to trade your attestation for more valuable service. We have been paying for the web by letting advertisers correlate our behaviors together for a couple decades now.
Why did you need it from a script? Right click -> Save Page As... is still a thing and works quite well in my experience, even with complex React SPAs.
It's not just excess bandwidth. It's cpu cycles, poor site coding, database issues. Like I said, not everything is backed by a huge company that has the money/hardware/engineering resources to absorb the load.
That’s the point. For every legitimate though ultimately unimportant use case like yours there are a thousand others who would scrape the results 86,400 times a day to either try to sell or mirror and plaster with their own ads.
And those bot protection mechanisms and ad-enforcement layers that terrorize humans are okay or what? Yes, I accept that many pages don't want me there and just don't use them anymore, but it sucks.
I am not saying that forks like this are a good idea, but I have enough frustration with said techniques that I sympathize with the effort.
Man, this sucks. I doubt there’s anything that I can say to get people to stop doing things like this, but the eventual outcome here isn’t going to be freedom for you to scrape sites that are trying to avoid being DDoSed by bots, but instead that we all end up in a world where device attestation is required to do practically anything online. And for what?
Is it ethical to scrape when a site has explicitly blocked bots? I know a fair number of people who run small sites who are already considering closing them down because the bots are relentlessly hammering their sites and driving up hosting costs.
Use cases range from sending 100s of requests per second just to bring a website down to doing a montly request to a municipality's endpoint to get a local dashboard of when trash is picked up. I don't think you can pass a single judgement of automating web requests in general.
> Is it ethical to scrape when a site has explicitly blocked bots
Ethical? ppl get jailed in China for this. "Breaching computer systems" is a felony.
I know for a fact companies in China do this. One paid me directly for work in this space, and in another, I found my own OSS project in their artifact cache while working there.
I've only ever gotten in trouble in the UK
I think it really depends on how the bot is run. If the bot is replacing me navigating there manually, absolutely. If it’s sucking up content to rip off and make someone else billions of dollars, no.
Why do you feel entitled to navigate to a website you neither own nor pay for via a method the owner expressly forbids?
Why do you feel entitled to dictate what user agent I use, if it’s well-behaved?
Why do your opinions about what a "well-behaved" agent is override the wishes of a site's owner?
Site owners should not have full control over what users do. Their wishes do not apply past certain boundaries.
For a non-AI example I run into occasionally, any wish to stop time-shifting should not be relevant to what I choose to do as a media consumer.
There actually is a well defined demarcation point where the site owners wishes do apply. It's called... A demarc point! Everything that happens past this point is by definition entirely in the control of the admin. Which includes closing the door on whoever they wish. This is a technical reality, not something you can effect with opinions or wanting things more than other people want other things. It might seem contradictory but this is actually the main thing that makes the internet as free as it is.
When we're talking about ethics, I think we should give the owner more than that. If they don't want to be DDoSed we shouldn't say "too bad, the attackers are outside your network, your wishes don't apply, get good".
A request is a request. None of your business what I do with it after that.
If it's just impersonating me to help me better consume the content as if I were the one driving, then it's perfectly ethical, and not even related to a bot ban. Shades of gray and all...
Because I should be able to choose whatever user agent I want. The owner gets to "forbid" things on his computer, not on mine.
This page is only viewable in Internet Explorer 5.5. Please switch to a supported browser.
Accessibility, saving time, personal preference. Any number of reasonable things that don’t put undue stress on the host
It's most definitely unethical.
I think that instead of trying to prevent web scraping, websites should try to make it easier so that it generates less traffic. As long as any user is allowed to view the website, there will always be a way to scrape it anyway. If there were simply a monthly updated torrent available on a standardized subpage, such as example.com/scrape, scraping would be much less harmful.
> Bot detectors flag automation by reading the browser fingerprint; Fortress corrects that fingerprint inside Chromium's C++, so the browser presents as an ordinary Chrome install.
This does not seem like it would work against anything but the most basic bot protection.
[dead]
[dead]
For agent/browser automation, getting blocked is only one part of the problem. The other hard part is knowing whether the page you got back is the real page, a degraded version, or some silent challenge page.
odd name; shouldn't it be named after something that stealthily infiltrates fortresses?
Trojan Horse probably wouldn't make a very good name for a browser.
Since my grade-school mascot/team was "The Trojans" and my adopted hometown is firmly rooted in pre-Christian Greek culture and mythology, I am currently studying the Trojan War, and I may observe that "Trojan" is an extremely perspicacious brand-name for condoms, more than the average guy would expect.
Trojan Condoms: We will “trick” the defenses 100% of the time. Wait wrong take.
which is exactly the problem, a condom is famous for stopping things getting through and we do the literal opposite.
Actually condoms are famous for being quite unreliable and failing.
People using them are often uninformed and inept, using them incorrectly and hastily.
This is exactly why they are distributed with gusto by certain services that profit from “surprise pregnancy”.
fellow contributor on this.
we thought of it the other way round: your automation is the fortress, and every bot-detector trying to fingerprint it is the siege.
also most good burglar names were taken on PyPI
in my defence, i did lobby hard for "GuyWhoWavesYouThroughTheGate"
"Doorman"
I think more like mimicking the perfect resident. Then again it’s Theseus’s ship no, if you are able to do something so perfectly what is the difference between you and the clone? A Hollywood movie in the making.
"Of course... Why didn't I think of that?"
Algae
Is it possible to give negative stars to a GitHub repo? This one would be a prime candidate for that feature.
Does Anubis still work? (against this?)
The authors are proud of facilitating the theft of intellectual property. Disgusting.
Things like this make me wish that we have to pass ethics courses to work in tech.
Reminds me of this old joke: No ethically trained software engineer would ever write a destroyBaghdad function - they’d write a destroyCity function to which you could pass Baghdad as a parameter.
This is really unacceptable folks. There are those of us that have to keep these sites up, and it's seriously been a few years of nightmare scrapers and botnets, and stupid things like this that you are trying to legitimize that will make this worse. If a site doesn't want you, you should go away. There's a reason for it. Not every website is backed by a billion/trillion dollar company with the resources to absorb things.
Making a network request from a script is not abuse, consuming excess bandwidth is. Scrapers already spoof browser reputation and cycle IPs while abusing bandwidth. Recently I wanted to convert the results of an Autotrader filter into a table so I could supplement it with data they don’t track (towing capacity). It was two pages of results, but requesting it from a script was aggressively blocked by browser fingerprinting. I had to port it to JS and run it in my browser console manually to get the data out, wasting my time.
I think this is the point that has become a moral dilemma. Increasingly, in this agentic age tasks like research and web exploration that would be done by humans and even buy things from an website is now done through agents. While I sympathize with the main author about Ddos, but blocking every agent regardless of intent seems cruel. This would allow an MCP doing deep research(almost identical to human behavior) to move forward. But there should definitely be more work done system that focus on agent intent(i.e some agent trying to find the best price for vacation or scraper building alternatives to Travel Advisor).
Ideally, there would be a better way for web hosts to block bandwidth-hogging, but not block bots. I don't think anyone cares if some automated user-agent requests a 1KB file from their web site. They care that automated systems are sucking down TBs per day, all day, every day.
Making people actually read content doesn't strike me as cruel. Tedious, sure, but not cruel. What is cruel is seeing all of our collective individuality and artistic expression jammed into an LLM, sold back to us by the token, and forced into our lives by an economy increasingly skewed towards holders of capital.
Fair point. Again, I might be terribly and almost Icarus-like optimistic, but I still believe the push and pull between Big Tech and Indie AI builders would still lead to an equilibrium that makes the world a little more efficient and perhaps in time, more creative than AI slop. Let's hope Open Source AI opens instead of OpenAI so the internet is not feeding the capitalist machine of OpenAI and Anthropic.
CDN security SWE here. I have functionally zero interest in what individual people do in terms of automation. I automated my gym's class signup forum during COVID so I get the validity of the use case. You want to work around the mitigations to do the same? Go with my blessings.
The killer is that everything that works for individuals trying to get through the day and make the web a bit smoother is immediately used by industrial crawlers strip mining the Internet, and you can't block one without blocking the other. A future where the web is only accessible via device attestation is extremely dystopian but so is an Internet where every drop of content goes into an LLM training set.
Things like this is why all of the worthwhile content is going to drain into balkanized spaces, and we're all the poorer for it.
Hmm, I am sure you know more than I do on the topic. At least to me, an amateur in the security space, would the action of the scraper not be closer to repetitive calls to tables instead of, i.e a more inquisitive agent doing research or booking who spends more time looking through, and surely that can be tracked? Again, on the moral side, I agree that if one comes with the other, it is quite dystopian. I myself am an ML Inference Engineer, so I guess interesting problems and interesting solutions always draw me in much like this.
Abusive crawlers use residential proxies to distribute crawl traffic between thousands of IPs, so that one agent that's making five requests blends in with the massive crawl. There is signal but there are ML inference engineers tuning their traffic to blend in, which requires that mitigations must get more sensitive.
At the end of the day, the defensive side has an obligation to protect customers and well behaved agents are the first thing that gets swept up along with the bad actors.
It’s not that dystopian if you are given the choice to trade your attestation for more valuable service. We have been paying for the web by letting advertisers correlate our behaviors together for a couple decades now.
The concern is that attestation turns into another mechanism to monitor and control people; see how age verification turns into surveillance.
Why did you need it from a script? Right click -> Save Page As... is still a thing and works quite well in my experience, even with complex React SPAs.
It's not just excess bandwidth. It's cpu cycles, poor site coding, database issues. Like I said, not everything is backed by a huge company that has the money/hardware/engineering resources to absorb the load.
That’s the point. For every legitimate though ultimately unimportant use case like yours there are a thousand others who would scrape the results 86,400 times a day to either try to sell or mirror and plaster with their own ads.
And those bot protection mechanisms and ad-enforcement layers that terrorize humans are okay or what? Yes, I accept that many pages don't want me there and just don't use them anymore, but it sucks.
I am not saying that forks like this are a good idea, but I have enough frustration with said techniques that I sympathize with the effort.
Sites that are backed by a billion dollar company typically have better antibot protection, and lot of expertise in this.
Also, patched browsers have existed since long ago, although they were not open-source.
Unconscionable.
[flagged]
[flagged]