Once I was at a company which both had agreements to rehost certain content, and scrape the significant amount.
A couple of funny things happened, once we were clearly being blocked and my manager wanted me to figure out a way to evade said blocking.
And in another case we ended up getting scraped ourselves. Which was absolutely hilarious. I think we were able to resolve this by just contacting the website host.
I am not a lawyer, but a lot of things in America are almost 100% dependent on how much money you want to spend suing or defending yourself in court.
There are a lot of VC funded companies who more or less just scrape and resell data. If LinkedIn and few other places figure out how to actually block them , they'd be out if a business.
The recent DOSs are not scraping. You can scrape a website running on a hobbyist home server without inconveniencing it, even 1000 people daily can. The AI "scrapers" sending multiple requests per second (even per day) to every imaginable URL (vs. sitemap.xml or robots.txt) should be more accurately called DOSers, and are intentionally malicious and/or ridiculously incompetent.
Maybe you should not generalize your own knowledge to "in other countries". I no country where I have lived do you get fined for a measured speed slightly above the posted speed limit. The difference may be mainly there to account for measuring imprecision but effectively it means you can get away with a bit over the speed limit.
Yes, there is an "enough or often" limit in the USA as well. State driver's licenses are based on a point system and too many infractions will result in a suspension of your license if your points dip below a given threshold. Your margin of speed vs limit increases the point deduction. Fines vary down to individual localities, some of them being annoyingly high. In most places (I believe) driving with a suspended license is a criminal offense.
One big difference may be that most places in the USA have been able to resist photo radar/LPR automation of issuing speeding tickets, so the odds generally favor you in most areas until they don't. There's also this tacit allowance for "going with the speed of surrounding traffic". If all the traffic on the freeway is going 10MPH over the limit, you're generally safe in the herd like a zebra.
The USA is a big place with a very diverse set of driving conditions and jurisdictions, so the more you know an area the better you can navigate the character of that area's enforcement. Some places you'll deal with flagrant enforcement pushes at the end of the month as police try to meet quotas. Some places use traffic law enforcement to subsidize their budgets. Our mapping apps can alert us to speed traps if other drivers report them. Drive an area long enough, you'll know where the common speed traps are, and understand the patterns in road design that facilitate them. Generally, we have the luxury of being able to flout speeding laws to such a degree that we can feel indignant when we're caught, but the law is the law.
Well, as he goes on to say, a scraping campaign must (based on current law, anyway) satisfy a fairly specific list of constraints to be considered unambiguously legal. That's where the "not clearly legal" idea comes in; I can only imagine that law students would have trouble parsing the idea that a massive tech company with billions of dollars would fail to spend even a little bit of that to evaluate whether or not their scraping was legal and just... do it anyway.
Honestly interesting article and general topic. As with most things it depends on who you are. If you are Spotify, Google, OpenAI or Anthropic, its fine. As mentioned, this is why gray areas and unsettled laws are so malicious. They protect the rich and powerful and can be used to absolutely destroy regular people. It is adjacent to scraping, but Aaron Schwartz is a good example. He faced decades in jail for what is effectively best practice for an LLM data pipeline
> It is adjacent to scraping, but Aaron Schwartz is a good example. He faced decades in jail for what is effectively best practice for an LLM data pipeline.
None of what Swartz was charged with [1] is anywhere near best practices for an LLM data pipeline.
(Also, he wasn't facing decades in jail. That's from the DoJ press release which uses a completely stupid way of calculating potential sentences [2] that gives very high numbers so it always make it sound like they have caught some super villain).
I don't think that there are laws against LLM distillation in the United States. It's only a terms of service violation. For all of the complaints about model distillation that OpenAI and Anthropic make, you don't see them saying that it's against the law:
They would probably like it to be against the law ("Distillation attacks at this scale require a coordinated response across the AI industry, cloud providers, and policymakers") but I hope that legislators decline to grant their wish.
Distillation is legal. ToS violations are not crimes. There's nothing illegal about capturing output to train with it.
There will probably be (hypocritical) AI companies trying to claim some kind of minimal "protections" are applied to the output in order to misuse DMCA violation claims, but it's BS.
Once I was at a company which both had agreements to rehost certain content, and scrape the significant amount.
A couple of funny things happened, once we were clearly being blocked and my manager wanted me to figure out a way to evade said blocking.
And in another case we ended up getting scraped ourselves. Which was absolutely hilarious. I think we were able to resolve this by just contacting the website host.
I am not a lawyer, but a lot of things in America are almost 100% dependent on how much money you want to spend suing or defending yourself in court.
There are a lot of VC funded companies who more or less just scrape and resell data. If LinkedIn and few other places figure out how to actually block them , they'd be out if a business.
The recent DOSs are not scraping. You can scrape a website running on a hobbyist home server without inconveniencing it, even 1000 people daily can. The AI "scrapers" sending multiple requests per second (even per day) to every imaginable URL (vs. sitemap.xml or robots.txt) should be more accurately called DOSers, and are intentionally malicious and/or ridiculously incompetent.
This one made me chuckle:
> This breaks my students’ brains. If it’s so ubiquitous, it must be clearly legal…right?
Do they not drive? Driving slightly faster than the speed limit is ubiquitous in my experience.
I'll guess you are from living in the USA
In other countries going over the speed limit a small amount will get you fined and your license suspended if you do it by enough or often.
People in countries like Australia have to warn US visitors that they have to keep below the speed limit
Maybe you should not generalize your own knowledge to "in other countries". I no country where I have lived do you get fined for a measured speed slightly above the posted speed limit. The difference may be mainly there to account for measuring imprecision but effectively it means you can get away with a bit over the speed limit.
> "enough or often"
Yes, there is an "enough or often" limit in the USA as well. State driver's licenses are based on a point system and too many infractions will result in a suspension of your license if your points dip below a given threshold. Your margin of speed vs limit increases the point deduction. Fines vary down to individual localities, some of them being annoyingly high. In most places (I believe) driving with a suspended license is a criminal offense.
One big difference may be that most places in the USA have been able to resist photo radar/LPR automation of issuing speeding tickets, so the odds generally favor you in most areas until they don't. There's also this tacit allowance for "going with the speed of surrounding traffic". If all the traffic on the freeway is going 10MPH over the limit, you're generally safe in the herd like a zebra.
The USA is a big place with a very diverse set of driving conditions and jurisdictions, so the more you know an area the better you can navigate the character of that area's enforcement. Some places you'll deal with flagrant enforcement pushes at the end of the month as police try to meet quotas. Some places use traffic law enforcement to subsidize their budgets. Our mapping apps can alert us to speed traps if other drivers report them. Drive an area long enough, you'll know where the common speed traps are, and understand the patterns in road design that facilitate them. Generally, we have the luxury of being able to flout speeding laws to such a degree that we can feel indignant when we're caught, but the law is the law.
Well, as he goes on to say, a scraping campaign must (based on current law, anyway) satisfy a fairly specific list of constraints to be considered unambiguously legal. That's where the "not clearly legal" idea comes in; I can only imagine that law students would have trouble parsing the idea that a massive tech company with billions of dollars would fail to spend even a little bit of that to evaluate whether or not their scraping was legal and just... do it anyway.
This is a feature not a bug in the system. But for a lot of idealistic young law students learning that will make them deeply uncomfortable
Ask a cop pulling you over if it's a feature as he writes you a ticket lmao.
Honestly interesting article and general topic. As with most things it depends on who you are. If you are Spotify, Google, OpenAI or Anthropic, its fine. As mentioned, this is why gray areas and unsettled laws are so malicious. They protect the rich and powerful and can be used to absolutely destroy regular people. It is adjacent to scraping, but Aaron Schwartz is a good example. He faced decades in jail for what is effectively best practice for an LLM data pipeline
> It is adjacent to scraping, but Aaron Schwartz is a good example. He faced decades in jail for what is effectively best practice for an LLM data pipeline.
None of what Swartz was charged with [1] is anywhere near best practices for an LLM data pipeline.
(Also, he wasn't facing decades in jail. That's from the DoJ press release which uses a completely stupid way of calculating potential sentences [2] that gives very high numbers so it always make it sound like they have caught some super villain).
[1] https://volokh.com/2013/01/14/aaron-swartz-charges/
[2] https://web.archive.org/web/20130211030902/https://www.popeh...
If scraping is legal, so should distillation
I don't think that there are laws against LLM distillation in the United States. It's only a terms of service violation. For all of the complaints about model distillation that OpenAI and Anthropic make, you don't see them saying that it's against the law:
https://openai.com/index/disrupting-a-coordinated-model-dist...
https://www.anthropic.com/news/detecting-and-preventing-dist...
They would probably like it to be against the law ("Distillation attacks at this scale require a coordinated response across the AI industry, cloud providers, and policymakers") but I hope that legislators decline to grant their wish.
I don't see why that should necessarily be true.
If you read the article, scraping is “not clearly legal”. Read those 3 words carefully.
Distillation is legal. ToS violations are not crimes. There's nothing illegal about capturing output to train with it.
There will probably be (hypocritical) AI companies trying to claim some kind of minimal "protections" are applied to the output in order to misuse DMCA violation claims, but it's BS.
[flagged]
[flagged]
[flagged]
[flagged]
[dead]