It’s also a “you can’t scrape us but we can scrape you” mentality these companies are getting into.
It really sucks for small businesses. Their websites get absolutely hammered and they can’t afford or don’t know how to stop it. Cloudflare ends up winning as they’re the easiest solution.
It works similar to cloudflare WAF but is an open source software. It can be on your own infrastructures even if its a $5 VPS.
The idea is same let the bot or AI agent do more work than it takes to serve that request. Also it needs to follow rules, similar to cloudflare enterprise web application firewalls. It was OWASP CRS and J4A TLS support.
Yes name was anubis backwards and thats the only resemblance between the 2 projects and also I used anubis, safeline for benchmarks compparison. Each of the design records are in the repo if you want to see the evolution and algorithms used.
Its not a rewrite and is more inspired by https://github.com/chaitin/safeline. Wanted a web interface and a distributed waf similar to cloudflare as an alternative. It supports OWASP CRS, J4A TLS trrmination handling. Comes with console-ui and also the proof of work used a different algorithm which is proven to be resistant to quantum attacks.
It's a cognitive tax on all the parties involved. It works on buyers (websites that pay for it) much the way that llm's work on their users, by working up to the point at which the buyers can't discern it working.
As a user (bot or flesh) you have to jump through these hoops, and to the website operators they get fewer complaints when users (and bots) are able to contort themselves in predictable ways. Versus you'd perceive as a website operator, if your blocking worked at all, that you'd risk seeing a precipitous dropoff of bots if you did things in bespoke ways, as both bots (true positives, among dd and cf survivors) and people (false positives) get jammed.
The net effect in this model would be that such things as cloudflare and datadome take money from websites in order to block "dumb" bots and enfranchise "smart" bots to the point of creating a market for bot services. People that want to bot have to use AI or go to bot service providers.
For website operators that's a worthwhile exchange for dollars, in that dumb bots can cause a lot of harm in wasted compute. Bots are typically dead end users who won't buy ever anyway, or merely inefficient way of doing business if they scrape excessively. And many do scrape excessively.
"reform school for bots" is a good mental model. For this to work, the "reforms" have to be consistent so that bots can conform to them.
It's a peculiar equilibrium and I am not sure that it's all that stable as an equilibrium. Would be a good subject for a formal treatment, game theory and economics.
Mostly economies of scale. They have enough data to do the statistics to identify bad actors, and enough bandwidth and servers to facetank the request volume.
> It’s also a “you can’t scrape us but we can scrape you” mentality these companies are getting into.
And the problem often isn't some advanced attack. It's enough to follow links with session params, generate endless urls or ignore robots.txt to turn a regular crawler into a load generator. A site owner needs crawler protection from specific behavior and not another list of known bot signatures
Cloudflare doesn't even stop bots very well. Many bots are using real browsers, which pass all fingerprint checks, and scripts that can click the checkbox. What Cloudflare has is mindshare, and being a checklist item and an accountability sink.
Does it suck if you offer a product and someone’s looking for it and the agent can’t look on Amazon but they can look on your small side maybe that’s good
Legit what one of our clients problem was. Normal web traffic wasn’t an issue and the logs were just a few megs. Yeah there was a problem where they were not rotating but once meta started crawling them 24/7 ignoring the robots.txt it kept filling their vps up. An annoying problem to fix once a server is completely full and you can’t even ssh in.
There are also some pathological bad use cases out there that are caused by the combination of trying to scrape all the content, a randomized param, and the scrapers trying to be anonymous. The one I've encountered is pbpBB.
phpBB will assign you a randomized session if if you visit it anonymously (without logging in), and will serve the page to you with that sid param added to all links. Using that link will load a page without the sid param appended to links, because the system assumes you have a session I'd saved in a cookie since it saw you have a sid, even I'd you don't have/didn't send a cookie with it. Following any of those links starts the chain over if you aren't sending a cookie along with a sid, which the AI scrapers never do so they can't easily be blocked by that, and a new unique sid is generated, and all the links have the new sid appended.
Every page with a unique set of params and values needs to be investigated, because it can be an entirely different page, so AI scrapers will scrap forever, always finding more and more URLs to harvest, even though they're all duplicates of what they already have. It's easy to have multiple scrapers stuck going through a very small bulletin board continuously.
Maybe they've gotten smart enough to deal with this situation in the last few months, or maybe they put in a specific workaround. I ended up just throwing the site behind cloudflare because spending time trying to outsmart AI agents in a battle of escalation doesn't seem like an effective use of my time.
30GB a month isn't too terrible? I guess it depends - for example my provider includes 10TB egress per month, so 30gb would be negligible if it's static content. I'm not sure what others do, but I often click on the links that the agents pull up as sources so the websites do get more discoverable.
What really sucks is e.g. Gemini that pulls images without being able to go to the source (unless it changed recently.)
The issue is that a dynamic site that can serve the traffic that generates 10MB of logs a week might be very performant on the smallest and cheapest of virtual servers, but one that generated a GB daily might find itself very slow and sometimes crashing because of load depending on spikes, unless they opt for a server with more resources, for more money.
Any answer to that along the lines of "do something more complex" or "optimize your site for load" to be able to deal with the rampant bad behavior of multi-billion dollar companies is unpalatable to most.
If you think your only tool is a hammer, everything indeed starts to look like a nail.
This is one of those things that might be best solved through policy and law. There probably will be some changes to policy and law to help, eventually. That stuff usually takes a while, and that's always been a problem for those suffering in the interim.
Not every request is meant to be served below 500ms and this is server which supposed to be called couple of times a minute not milliseconds
We started with caching but this created a new kind of problems of its own (veterans know what circus of man eating from outside out clowns this create) but client was mad manual refresh is causing "freezes" in the system.
So we ban LLM and problem solved until client starts asking why ChatGPT, Claude, whatever he prefer can't access data and release him/her from pain of using brain
Cloudflare is probably one of the most popular, reasonably priced for general public. You have fastly, akamai, cloudfront as other proxy providers and CDNs with similar capability but higher price point mostly catering to enterprise customers.
Bunny.net is another one that I find similar to Cloudflare from pricing and functionality perspective
Depends on the small business obviously. I've been in IT for 40+ years and enjoy torturing clankers; I ran a bot motel 20 years ago. These things are so badly written, and obviously they don't care because they're not truly conscious, but the meat in charge is ignorant or doesn't care. So web scraping, and the point cloud of nearly indistiguishable abuse it travels with, is pretty easily tripped by some simple technical tests and salting of content. At least mostly.[0]
I've been wondering to myself "how can I get people to go through a login process without going through a login process?" for a few months, and I've come up with a simple solution which I'll be implementing in the next few weeks.[1] Like the other technical measures it doesn't identify bots but there will be a gate with backpressure that at a point in time a given IP address has some probability (adjusted in realtime based on $things) of passing through, and through my mysterious process is then "logged in". Then that address is pretty much given a free pass as long as its behavior remains "human". Real humans may be denied access initially and have to try again, and their luck will be affected by how aggressively the bots are fighting it out at that point in time. This isn't much different than the extant reality, but it is in terms of what happens after an initial successful page load. It's an experiment, I'm hoping it's tunable so that the experience is more or less indistinguishable from the status quo; we will see.
I'd love to find a group of likeminded individuals where we can discuss some of this TLP:YELLOW or Chatham House Rules. If you are savvy enough to hunt me down and can demonstrate that you're a real human, small operator, and that you can keep things closely held I'll talk with you, if you want someone to run your mitigations that would cost money.
[0] Had an episode last week where somebody scanned approximately 1000 URLs (maybe 60% of the discoverable URLs) over six hours. 1200 addresses fetched exactly one URL, another 400 or so fetched two or three. Every one of those URLs was fetched exactly two times. Real showshoe stuff, nicely done.
[1] Firewall rules and fail2ban. I give fail2ban special powers in terms of the logs it follows and the actions it can invoke. Conntrack without conntrack (of course the firewall has conntrack, it's just not the best option for this use case).
TikTok, Instagram, X, etc. These platforms will host you for free and will distribute updates to people, including potential customers via the algorithm.
Funny that I can't barely access any of those just because I don't have an account. Plus Facebook. They show me almost nothing and push their shitty app to nausea. And I am sure the "site" owners get zero info on how many clients they are losing for being behind that fence.
Pretty sure they're referring to their usage as a potential content consumer, not content creator. You don't need a cloudflare or AWS or any other service account (usually) to be an end user of a site that utilizes those. Increasingly it's hard to view any content on the social media platforms without an account.
Small businesses should be the people happiest about AI scraping. It's a new venue opening up for them to sell their products and services. And it's easy and free to use Cloudflare so that you don't have to worry about excess traffic, if you don't want to use a different solution.
This is expected. A lot of these companies that are closing their doors make a large portion of their revenue based off of ads and upsells which agents can easily be instructed to ignore, and don't interact with like a human.
Another more blunt way to say it is that people can instruct their agents to look out for their best interests (and while there are cracks they continue to get better at this), and companies hate that.
People will work around it similar to how we've worked around with adblockers for years, and it will just become a game of attrition as these companies work to preserve their moats until they find a way to preserve this revenue stream off of people's agents.
Yes, same with many other things where people were under informed or did t have the time to look out for their own interests. Such as filing complaints, citing the correct legal sections, knowing the procedures, knowing their rights, noticing when they are getting an unfair deal, etc etc.
I do see that there is another side to this. I saw complaints by an event organizer (think weddings etc) who got some special requests and went above and beyond to accommodate but it turns out the extensive requests likely were AI generated fully optimizing for the customer's interests, and just shamelessly asking for many favors essentially. She used to do these things because humans had a bit of embarrassment and friction around asking too many things and so it worked on a social agreement that if you ask it likely is truly important that you made the effort to ask. Now this cannot be assumed. So things will more to more formal rules, more explicit pricing for the small asks, and generally more adversarial and low trust systems that fully squeeze out the human slack from these systems.
Websites are closing their doors to their users too. I'm unable to register a GitHub account because the third-party bot protection service they use, called DataDome, thinks I am a bot just based on my IP and probably my extension combination, both in Firefox and Chromium. Datadome is also preventing me from accessing some regional sites of Leroy Merlin, Idealista, and a few other major websites in Portugal and Spain whose IT management suddenly decided Datadome is good for them.
A year ago, I contacted DataDome directly and they assured me that "fighting false-positives is their biggest priority" and their filters were fixed for a while until recently. A few days ago, I contacted them again, and now their stance is different: "our support channel is for customers only and this ticket will be closed." However, their customers have no idea that DataDome messes with their end-users, and reaching a tech support person who actually knows about their website using DataDome is unbelievably difficult.
Datadome reminds me of Cloudflare in their early days, when they were breaking UX for many users with Google reCAPTCHA.
I'm super interested in curating a dossier of these sites and experiences like yours, if only because we're genuinely interested in preventing issues like these and don't do the types of things that lead to these complaints, like IP-range profiling.
they would need to produce all data they have about you. It may not lead to anything but you can mess up with them as much as they are messing up with you.
That's perfectly fine by me. I'll read those arguments and buy those things my agents can find on the open web; which is plenty.
Want to argue for a shining new world, but you're on a locked site? I guess we'll have to live with dystopia. Want to sell the greatest widget since the invention of sliced bread? Well, I never found it. Sorry, I bought inferior widgets from your competitor instead. I'm at a physical sales location, but I can't access your spec-sheets? Your competitor the next shelf over doesn't have that problem, I'll go buy theirs.
Exactly. Shut yourself off, you get shut off. I don’t give a damn you want me working the mouse on your site. Your site probably sucks ass and loaded with ads and popup offers and cookie banners and login hoops other bullshit I can’t stand that wasted my time. Oh check my email for a one time code becuase you don’t like passkeys.
I hope the irony isn't lost on these companies. Meta is one of the most aggressive companies against any sort of automation, perhaps after LinkedIn.
I'm trying to build a personal social media client to stop the platforms from shoving their feeds down my throat and to consume only the content I care about, so this hits home.
(I just got blocked from LinkedIn for letting an agent set all of the 77 email notification toggles off, which in hindsight was obvious but I wasn't going to do that manually since every toggle has their own separate page, it's a lot of clicks.)
Pro-tip: it costs very, very few tokens to have an agent (harness) sleep a random amount of hours between each toggle. Just tell it to. It's not burning tokens on `sleep`, just to call it.
It might not hit input cache at that rate but better than a block or ban. Keep the context window small.
"Companies" are not capable of feeling irony. They absolutely will demand the unfettered right to scrape your stuff while demanding that your scrapers be outlawed at the same time, and if you go "hey, that's hypocritical of you", you won't get any response, not even a half-hearted rebuttal or uncaring shrug. Taking about "hypocrisy" or "philosophical consistency" with regard to corporate behavior is almost a category error, these aren't concepts which even exist in that space.
Companies don't want it to be easy to cross shop their competition, the whole premise of Amazon is that it's so easy and simple you don't even check the competition. AI agents are leveling the playing field and helping customers find the best deals, so of course they're being banned
Everyone focuses on Amazon here, but there is an entire industry of e-commerce tools and products and platforms and services designed around optimizing e-commerce sites for sales and other uses.
Agents invalidate a lot of that and so it is an existential threat to many of them. I expect the near future will result in a lot of different results tailored to agents that may actually result in higher priced products for agents browsing versus humans.
Recently bought something online from Home Depot and it arrived damaged. When I asked them for help they said I had to drive it back to the store myself.
If I have to drive to the store to get it, I wouldn’t have shopped with them! It feels like every time I don’t order with Amazon the stores remind me why I ought to.
Amazon's strength lies in their logistics/3PL/last-mile systems. They invested heavily into this very early on into their business and it's been paying off ever since.
Walmart is the only other company that can compete at that scale given their very mature ground distribution network and even then Amazon has them beat good on air cargo (Amazon has a virtual cargo airline operated by Air Transport/Atlas Air, both of whom they have some ownership over) and inventory management (they hired some very very smart folks to help them build inventory forecasting systems)
As long as Amazon remains the only company that you can buy almost anything and get it in two or three days almost anywhere on Earth, they won't need to worry about the grim reaper coming for them.
It's not that hard. Make an iOS app and use the remote attestation features to verify the client. In theory Android has similar mechanisms available, not sure how secure they are but Apple's is pretty good.
Now whether you want to restrict your shop to only being accessed via a mobile app is a totally different proposition.
An easy solution is to sell a "web search" subscription add-on that gets distributed to the sites your agents utilize. The script will flip overnight and ASO (agent search optimization) will be born. It could be a form of micropayment system that the internet has desperately needed for over a decade now.
Without a convenience moat, commerce will get even more adversarial: membership pricing, surveillance pricing, loyalty perks, varied products to avoid price comparison (like mattresses and Costco).
Fortunately, if AI is really superhuman in capabilities it may still win the battle for the customer and keep prices competitive: aggressive price comparison, alternatives discovery, negotiation, group buying, etc.
How would the agent know it's a better product? All it has to go on are whatever bot-generated spam reviews and prompt injections that they can scrape.
In a fully agentic shopping experience, it's not obvious to me that providing a better product works to drive sales. Bargain-basement stores can make up a story about how their goods are lovingly handcrafted for decades of durability (adversarially tested against the most common agents, so they'll believe it) much more efficiently than I can honestly provide one.
First they forced their malware bot farms on us, now they want to rent them to us so they can turn the entire web into the dark web.
And eventually rug pull off the major retail sites and just redirect traffic to their own darker sites to scrape commissions, sell direct and keep the change.
I'm sure the teams thinking this up buy their lunches from a ghost kitchen run by illegals that's never had a health inspection,
located in the back of a laundromat with a phony business license and website that only accepts crypto.
I think people who rally against Amazon in this topic assume that the "savings" from much open price competition immediately come back to the consumer or the manufacturer. In my experience, with previous such tech improvement such as 'uber disrupting taxies', this has never been the case.
Both the consumer and the manufacturer/labor/driver/gig-worker always get the short end of the stick.... once AI-agents becomes indispensable part of life.
Uber is a single centralized platform that sets prices based on their secret algorithm to maximize profit, gives worse deals to drivers who look desperate and shows different amounts to two riders standing next to each other. It's as far from real competition as you can get.
It would be way different if it was an open network where drivers are allowed to set prices and reject rides without any penalty, and riders can send multiple offers at a time then take the best one. That would be a real competitive market where everyone is free to choose what they buy and sell.
In that scenario Uber would be one of many apps you can use that index the network, whitelist trusted drivers and handle payments for a small fee. And if you want to make a new app or do the whole process manually there's nothing to stop you. That's what shopping on the open web is like.
I am not disagreeing on any of those points. Just pointing it out that tech disruption only happens when someone else finds a way to steal that cut. It is never about the common good. That is why true open solutions and markets are hard to find.
I would be surprised if you had enough data to be able to conclude such a lofty claim, especially accounting for increased availability of taxis in areas where there were none or few.
Either way, the benefits of price transparency are not imaginary, but they are also not the only factor. Things like insufficient buyers or sellers, government policy, etc are also in play.
All else being equal, a buyer being able to easily see seller A is more expensive than seller B for the same goods will result in lower prices for the buyer. But all else is rarely equal.
it looks like this is solving itself in the usual and productive way:
> Instead of trying to fight Amazon, the company has partnered with retailers like Walmart, Gap, Sephora, Best Buy, and others, allowing Muse to shop on their sites. At the same time, Shopify has also opened its network of storefronts to AI agents.
they're making deals with the sites so that the agents can work, the sites get compensated. I hope for this "AI can do what it wants if it pays" model to expand further. Cloudflare told us months ago we'd have an option to get paid by AI crawlers. I signed up to join but of course as a nobody I never got a reply.
That's how the crawler deals work (Reddit licensing its data, Cloudflare's pay-per-crawl). For shopping agents, Meta has described the money going the other way. Zuckerberg on the Sources podcast in September: the plan for Muse is "a very small cut of whatever the transaction is… It'll come from the businesses that they're working with." Shopify currently tells merchants there's no fee, so this is the plan, not the present.
So the stores aren't being paid to let the agent in. Meta expects them to pay it.
This is just because agents don’t see ads right? Regardless, this is one of the advantages of software heterogeneity. The cost to detect rises substantially and the false positive rate will eat you alive. My agent buys me things. On Amazon too, yeah. OpenTable is hard to handle but I’m sure it’s fixable.
I will always block ads no matter what. I don't care about what is sustained, there will be some way to get these things, and if not, we will get used to that too. Meanwhile I will not tolerate ads.
Indeed. One could reasonably argue that viewing ads on Amazon plausibly lowers the cost of goods served, just as connected TV targeted advertising allows very low cost TVs. I think that’s quite believable. Given that, I’m a free rider, somewhat like a software or media pirate.
Not outlandish. The prices are a big motivator and I wonder how much they’d have to charge me to compensate. My agent probably sees the sponsored results before it sees the other results but that is probably considered insufficient compensation.
I think I’ll keep doing what I’m doing but primarily because if the world is forced to charge higher prices to compensate for the lack of advertising I think that isn’t an outcome I particularly am upset by. It passes a Categorical Imperative Test, if you will.
I've never updated on this: The web was better when there were little or no ad-revenue based services. (and I'm ok with actually paying for services that add value)
It's the latest iteration of "a little X is good, a lot of X is bad".
Email was awesome when it first became widespread. I could cold-email random professors at other universities (I was in school at the time) and get meaningful responses back. Then lots of people (and companies) started cold-emailing, and it stopped working.
Social media is broken if there's nobody there, is great with a medium-sized community, and is broken if everyone is there.
Ad-supported websites can offer all kinds of valuable free services. Up until either (1) they become so ad-infested that they're no longer useful, or (2) all the users block ads.
There was a wave of websites providing APIs for programmatic access a while back. You could script together whatever flow you wanted. If no API was offered, you could scrape the HTML. That triggered a wave of alternate interfaces to sites, skipping their ads, skipping their engagement funnels, separating the value from the cost. So everything got locked up behind captchas and accounts and API keys with limited access.
This outcome was always inevitable. Humans don't understand social contracts, or they do and both sides abuse the hell out of it. We feel entitled to have just the loss leaders without ever buying anything else. When running businesses, we feel entitled to keep adding more and more adds and generally enshittifying the experience. People can take it or leave it, after all.
Agents don't change the underlying structure at all. They just magnify and accelerate parts of the system, and will therefore magnify and accelerate the reactive and defensive responses.
A social contract towards a profit-maximizing corporation is not real. They are amoral entities with only one goal. The real implicit contract between you and a business is simple: they will do everything they can to make you give them money, and you will do everything you can to not give them money, or else you're a sucker. "Everything they can do" includes convincing you there's a social contract involved.
The whole web is based on selling people's attention, providing free services as "payment" to people. The whole value proposition of AI is to give the services without giving people's attention?
I have yet to meet a website I can't crawl with Chrome MCP. You just need to go slow and do extra actions that look human. When there's a captcha, AI notifies me to solve it, I do, it continues. All this is for personal use.
my prediction: ai agents will have their day, but not on this version of the web. websites will have a separate xml/json/yaml version that agents can peruse without all the ads/js/etc
My agents can just take that feed and render it into a custom front end just for me too. Why would I browse the adversarial, bloated, ad infested version?
Why would that happen if the website owners are explicitly trying to prevent access? This isn't a technical issue, the agents are perfectly capable of working with the existing websites regardless of ads and javascript. The semantic web and machine readable representations was a 90s thing, agents don't need that really.
no you are def right the agents are def capable and that is the problem, so capable we are all doing this, we are all making our own scrapers, as are big corps/small corps and it is adding up so now we are seeing managed challenges everywhere and webmasters/sysadmins going to great lengths to block any bots.
the js/yaml version would require a license key that connects to a business arrangement between the website and the owner of the crawler. the plain HTML version would adversarially block them (or even identify them, where they could be sued...once we have legal precedents for this stuff).
And finally, we will have the future SOAP and XML schemas promised us all those years ago!
Websites cost money. Websites won't cater to bots the same way they don't cater to the scrapers of old. Some bot companies may be charged extra to integrate their bots in some websites, though!
I agree. I’ve been saying for years that marketing teams need to switch to include marketing to LLMs.
I have (currently free but maybe paid in the future) accounts for side projects with businesses that Claude chose for me. (I reviewed the suggestion, but didn’t see a problem or reason to change it.) Why did Claude choose those providers? No idea, but if it’s through subtle marketing to Claude, then well done to them.
Without the ads? The pessimist in me thinks with all the ads, prompt injections, and long-form content marketing included. Already you can see Google's AI results rehashing marketing slop in the answers about how amazing one product or another is.
Wasn't there some restaurant review site that gets accused of doing pay-to-play hiding of negative reviews and promoting positive ones that calls up businesses and basically extorts them into their services? They have all the incentive in the world now to manipulate the results shown to AI agents. You won't even see the manipulation directly yourself when there's an AI sitting between you and the source acting like an obfuscation layer.
hm. for me, AI agents effectively already have their day unless there is a retrenchment. i essentially do not interact with the web (outside of a few social media sites) anymore
if the reason to do so was the ad revenue, then i definitely agree. but in terms of me happening on to your ideas, now is perhaps actually better than any time in the past. i also do pay for writing à la substack
Sheesh, that's hell of a plan. Insert yourself as the middleman for almost all internet traffic, push for a payment standard and then take a measly 1% as a fee. Boom, infinite money.
We should create a human knowledge base for AI to train on that is completly open so everyone had access. Not just big corporations. Then adopt Ethical Acceleration. https://youtu.be/nV3tBWnZ4wg?si=UX2lKXXPZr01FXYi
If only it would be possible to "close your doors to AI agents". It's not I'm afraid. Where do you draw the line? What browser is allowed?
For example, I now have to `pacman -S chromium` to buy tickets from Deutche Bahn. They accuse me of being a bot when I use firefox on linux. It's a known problem, and I think stuff like this will only get worse the longer this cat and mouse goes on.
We host a website in Canada (physical server hosting VMs in a Canadian DC) and with the ongoing trade-war we do whatever we can to avoid anything US-based tech such as Cloudflare. So far we use Anubis to protect our services from scraping and it's been a decent experience.
Weird time to do it IMO, right on the brink of agents potentially becoming the main entity closing purchases for many. If agents succeed at becoming e-commerce middlemen for the masses, I think these decisions will age poorly and there will be some walking back.
Entire networks. GPT dot became useless as it got systematically blocked by Cloudflare, Google cloud, vercel, etc. At the same time OpenAI blocked solutions like tailscale and others effectively killing the utility of browser control by dot.
Finally, we are at the dawn of the next wave of disruption.
Web 2.0 went from serving users to profit maximization. Amazon became unusable. Streaming services fractured and fragmented, with content locked behind numerous paywalls. Ads permeate every surface.
This is not to say the agentic era will end differently. But at the onset of this era, and for the foreseeable future, customer adoption and trust will decide winners and losers.
I mean, yeah; agents and AI training data spiders basically operate like DDoS swarms, and it's not like their harnesses implement client-side rate limiting or anything like that. WAFs were designed to block/slow exactly this kind of traffic, so make sense that businesses will use them as designed.
When people were uploading music videos on YouTube, media companies fought to the death to prevent that. Then suddenly they 180 and started promoting YouTube videos and begging for views.
This will go the same way, in 2 years 99% of Internet traffic will be AI agents. You close the doors, you stop existing.
Recently I've asked an agent to find me a laptop with particular requirements. You do not let the agent search your site, you will not get the sale.
This annoys me. I can understand stopping mass scraping by big labs, but if I tell my personal instance to try and find the best price for a 40" monitor, it should be able to get details from any site. It's acting as my proxy.
Not only this, but the AI will start picking up the scam websites that allow it to search it.
Solvable problems (just login or pay). This is more than just blocking anonymous scrapers. Sites are blocking access to agents even when they have a proper "ticket" for access.
It's insane to me that people don't see that agents are going to be how you interact with your computer going forward, almost exclusively. The people who make things hard for agents are going to get eviscerated by the people who don't just make it easy for agents to do things, but market primarily to agents rather than humans.
If that's true, why bother blocking the tools? Either they are useful in saving money (and more important, hassle), or they're not. If they're not useful, the problem simply won't materialize.
Because not blocking these tools causes additional operational costs.
For example, higher return rate for online shopping (because the tools didn't buy the correct thing, or maybe they bought something that they shouldn't have).
Or, the tool ignores the adverts on an ad-supported website, intended for humans, basically just adding costs for no benefit to the site.
Or, the tool wastes customer services' time on something that ultimately the agent's operator didn't even want in the first place.
It's a mistake to think the sites have a choice. I'm hardly going to trust an AI agent to shop on my behalf if it isn't smart enough to operate the browser on my behalf in a way that's indistinguishable from a human. They're almost good enough to do that now; it won't be long.
So the end result will be that the sites spend money on bandwidth to shove ads and dark patterns at me, which I will never, ever see.
You'd think they'd understand who's going to end up with the power to say "No" here, but apparently not.
It’s also a “you can’t scrape us but we can scrape you” mentality these companies are getting into.
It really sucks for small businesses. Their websites get absolutely hammered and they can’t afford or don’t know how to stop it. Cloudflare ends up winning as they’re the easiest solution.
In the spirit of open source thought about it and created sibuna (https://github.com/insanai/sibuna)
It works similar to cloudflare WAF but is an open source software. It can be on your own infrastructures even if its a $5 VPS.
The idea is same let the bot or AI agent do more work than it takes to serve that request. Also it needs to follow rules, similar to cloudflare enterprise web application firewalls. It was OWASP CRS and J4A TLS support.
What's the difference with https://github.com/TecharoHQ/anubis?
Considering sibuna it's anubis backwards.
Yes name was anubis backwards and thats the only resemblance between the 2 projects and also I used anubis, safeline for benchmarks compparison. Each of the design records are in the repo if you want to see the evolution and algorithms used.
Link here where they discuss Anubis https://github.com/insanai/sibuna#beside-anubis
Look like rust rewrite to make more performant?
Its not a rewrite and is more inspired by https://github.com/chaitin/safeline. Wanted a web interface and a distributed waf similar to cloudflare as an alternative. It supports OWASP CRS, J4A TLS trrmination handling. Comes with console-ui and also the proof of work used a different algorithm which is proven to be resistant to quantum attacks.
> Cloudflare ends up winning as they’re the easiest solution.
What prevents other hosting sites from providing similar tooling?
It's a cognitive tax on all the parties involved. It works on buyers (websites that pay for it) much the way that llm's work on their users, by working up to the point at which the buyers can't discern it working.
As a user (bot or flesh) you have to jump through these hoops, and to the website operators they get fewer complaints when users (and bots) are able to contort themselves in predictable ways. Versus you'd perceive as a website operator, if your blocking worked at all, that you'd risk seeing a precipitous dropoff of bots if you did things in bespoke ways, as both bots (true positives, among dd and cf survivors) and people (false positives) get jammed.
The net effect in this model would be that such things as cloudflare and datadome take money from websites in order to block "dumb" bots and enfranchise "smart" bots to the point of creating a market for bot services. People that want to bot have to use AI or go to bot service providers.
For website operators that's a worthwhile exchange for dollars, in that dumb bots can cause a lot of harm in wasted compute. Bots are typically dead end users who won't buy ever anyway, or merely inefficient way of doing business if they scrape excessively. And many do scrape excessively.
"reform school for bots" is a good mental model. For this to work, the "reforms" have to be consistent so that bots can conform to them.
It's a peculiar equilibrium and I am not sure that it's all that stable as an equilibrium. Would be a good subject for a formal treatment, game theory and economics.
Huh. I would expect even a nominal time or monetary cost would prevent spambots from being able to scale. That seems like a net good.
[dead]
Mostly economies of scale. They have enough data to do the statistics to identify bad actors, and enough bandwidth and servers to facetank the request volume.
Would love if they did but most end up just reselling cloudflare. Example WPEngine.
> It’s also a “you can’t scrape us but we can scrape you” mentality these companies are getting into.
And the problem often isn't some advanced attack. It's enough to follow links with session params, generate endless urls or ignore robots.txt to turn a regular crawler into a load generator. A site owner needs crawler protection from specific behavior and not another list of known bot signatures
Cloudflare doesn't even stop bots very well. Many bots are using real browsers, which pass all fingerprint checks, and scripts that can click the checkbox. What Cloudflare has is mindshare, and being a checklist item and an accountability sink.
Does it suck if you offer a product and someone’s looking for it and the agent can’t look on Amazon but they can look on your small side maybe that’s good
Not it the site can't stay up because it's constantly being hammered by bots.
Dude, I had a 1Gb of nginx logs daily then I blocked them and it dropped to 10mb weakly.
Legit what one of our clients problem was. Normal web traffic wasn’t an issue and the logs were just a few megs. Yeah there was a problem where they were not rotating but once meta started crawling them 24/7 ignoring the robots.txt it kept filling their vps up. An annoying problem to fix once a server is completely full and you can’t even ssh in.
There are also some pathological bad use cases out there that are caused by the combination of trying to scrape all the content, a randomized param, and the scrapers trying to be anonymous. The one I've encountered is pbpBB.
phpBB will assign you a randomized session if if you visit it anonymously (without logging in), and will serve the page to you with that sid param added to all links. Using that link will load a page without the sid param appended to links, because the system assumes you have a session I'd saved in a cookie since it saw you have a sid, even I'd you don't have/didn't send a cookie with it. Following any of those links starts the chain over if you aren't sending a cookie along with a sid, which the AI scrapers never do so they can't easily be blocked by that, and a new unique sid is generated, and all the links have the new sid appended.
Every page with a unique set of params and values needs to be investigated, because it can be an entirely different page, so AI scrapers will scrap forever, always finding more and more URLs to harvest, even though they're all duplicates of what they already have. It's easy to have multiple scrapers stuck going through a very small bulletin board continuously.
Maybe they've gotten smart enough to deal with this situation in the last few months, or maybe they put in a specific workaround. I ended up just throwing the site behind cloudflare because spending time trying to outsmart AI agents in a battle of escalation doesn't seem like an effective use of my time.
git hosts get the same effect - ddosers request a diff of every file between every combination of first and last commits.
30GB a month isn't too terrible? I guess it depends - for example my provider includes 10TB egress per month, so 30gb would be negligible if it's static content. I'm not sure what others do, but I often click on the links that the agents pull up as sources so the websites do get more discoverable.
What really sucks is e.g. Gemini that pulls images without being able to go to the source (unless it changed recently.)
Edit: I didn't read "logs"
The issue is that a dynamic site that can serve the traffic that generates 10MB of logs a week might be very performant on the smallest and cheapest of virtual servers, but one that generated a GB daily might find itself very slow and sometimes crashing because of load depending on spikes, unless they opt for a server with more resources, for more money.
Any answer to that along the lines of "do something more complex" or "optimize your site for load" to be able to deal with the rampant bad behavior of multi-billion dollar companies is unpalatable to most.
In other way I must use LLM to fix issue created by LLM in the first place
If you think your only tool is a hammer, everything indeed starts to look like a nail.
This is one of those things that might be best solved through policy and law. There probably will be some changes to policy and law to help, eventually. That stuff usually takes a while, and that's always been a problem for those suffering in the interim.
1GB of logs, not content
Average log size is about 300 bytes. 1GB of logs per day is approximately 40 requests per second average.
Not every request is meant to be served below 500ms and this is server which supposed to be called couple of times a minute not milliseconds
We started with caching but this created a new kind of problems of its own (veterans know what circus of man eating from outside out clowns this create) but client was mad manual refresh is causing "freezes" in the system.
So we ban LLM and problem solved until client starts asking why ChatGPT, Claude, whatever he prefer can't access data and release him/her from pain of using brain
Saying this to a group of people who hold ad-blocking in the highest regard is near peak irony.
what solution is there other than cloudflare that a small business (with decent dev knowledge and claude/codex) could use?
Cloudflare is probably one of the most popular, reasonably priced for general public. You have fastly, akamai, cloudfront as other proxy providers and CDNs with similar capability but higher price point mostly catering to enterprise customers.
Bunny.net is another one that I find similar to Cloudflare from pricing and functionality perspective
Make your site performance fast enough that bots don't bother you.
Depends on the small business obviously. I've been in IT for 40+ years and enjoy torturing clankers; I ran a bot motel 20 years ago. These things are so badly written, and obviously they don't care because they're not truly conscious, but the meat in charge is ignorant or doesn't care. So web scraping, and the point cloud of nearly indistiguishable abuse it travels with, is pretty easily tripped by some simple technical tests and salting of content. At least mostly.[0]
I've been wondering to myself "how can I get people to go through a login process without going through a login process?" for a few months, and I've come up with a simple solution which I'll be implementing in the next few weeks.[1] Like the other technical measures it doesn't identify bots but there will be a gate with backpressure that at a point in time a given IP address has some probability (adjusted in realtime based on $things) of passing through, and through my mysterious process is then "logged in". Then that address is pretty much given a free pass as long as its behavior remains "human". Real humans may be denied access initially and have to try again, and their luck will be affected by how aggressively the bots are fighting it out at that point in time. This isn't much different than the extant reality, but it is in terms of what happens after an initial successful page load. It's an experiment, I'm hoping it's tunable so that the experience is more or less indistinguishable from the status quo; we will see.
I'd love to find a group of likeminded individuals where we can discuss some of this TLP:YELLOW or Chatham House Rules. If you are savvy enough to hunt me down and can demonstrate that you're a real human, small operator, and that you can keep things closely held I'll talk with you, if you want someone to run your mitigations that would cost money.
[0] Had an episode last week where somebody scanned approximately 1000 URLs (maybe 60% of the discoverable URLs) over six hours. 1200 addresses fetched exactly one URL, another 400 or so fetched two or three. Every one of those URLs was fetched exactly two times. Real showshoe stuff, nicely done.
[1] Firewall rules and fail2ban. I give fail2ban special powers in terms of the logs it follows and the actions it can invoke. Conntrack without conntrack (of course the firewall has conntrack, it's just not the best option for this use case).
TikTok, Instagram, X, etc. These platforms will host you for free and will distribute updates to people, including potential customers via the algorithm.
Those are no longer useful as a "public square" because of the increasingly walled garden design.
Funny that I can't barely access any of those just because I don't have an account. Plus Facebook. They show me almost nothing and push their shitty app to nausea. And I am sure the "site" owners get zero info on how many clients they are losing for being behind that fence.
Then make an account. Billions of people have already done so. Cloudflare, or AWS, or any other server host also requires an account too.
Pretty sure they're referring to their usage as a potential content consumer, not content creator. You don't need a cloudflare or AWS or any other service account (usually) to be an end user of a site that utilizes those. Increasingly it's hard to view any content on the social media platforms without an account.
I'm not sure just skipping the scraping by providing your data directly for harvesting is the solution
Okay real quick, how do I configure a reverse proxy to my application with TikTok or Instagram?
You might be overthinking what is actually needed for the business.
> TikTok, Instagram, X
Requires login
[flagged]
Small businesses should be the people happiest about AI scraping. It's a new venue opening up for them to sell their products and services. And it's easy and free to use Cloudflare so that you don't have to worry about excess traffic, if you don't want to use a different solution.
This is expected. A lot of these companies that are closing their doors make a large portion of their revenue based off of ads and upsells which agents can easily be instructed to ignore, and don't interact with like a human.
Another more blunt way to say it is that people can instruct their agents to look out for their best interests (and while there are cracks they continue to get better at this), and companies hate that.
People will work around it similar to how we've worked around with adblockers for years, and it will just become a game of attrition as these companies work to preserve their moats until they find a way to preserve this revenue stream off of people's agents.
Yes, same with many other things where people were under informed or did t have the time to look out for their own interests. Such as filing complaints, citing the correct legal sections, knowing the procedures, knowing their rights, noticing when they are getting an unfair deal, etc etc.
I do see that there is another side to this. I saw complaints by an event organizer (think weddings etc) who got some special requests and went above and beyond to accommodate but it turns out the extensive requests likely were AI generated fully optimizing for the customer's interests, and just shamelessly asking for many favors essentially. She used to do these things because humans had a bit of embarrassment and friction around asking too many things and so it worked on a social agreement that if you ask it likely is truly important that you made the effort to ask. Now this cannot be assumed. So things will more to more formal rules, more explicit pricing for the small asks, and generally more adversarial and low trust systems that fully squeeze out the human slack from these systems.
Parasites love human embarrassment and friction. I'm all for more formal rules and explicit pricing.
Websites are closing their doors to their users too. I'm unable to register a GitHub account because the third-party bot protection service they use, called DataDome, thinks I am a bot just based on my IP and probably my extension combination, both in Firefox and Chromium. Datadome is also preventing me from accessing some regional sites of Leroy Merlin, Idealista, and a few other major websites in Portugal and Spain whose IT management suddenly decided Datadome is good for them.
A year ago, I contacted DataDome directly and they assured me that "fighting false-positives is their biggest priority" and their filters were fixed for a while until recently. A few days ago, I contacted them again, and now their stance is different: "our support channel is for customers only and this ticket will be closed." However, their customers have no idea that DataDome messes with their end-users, and reaching a tech support person who actually knows about their website using DataDome is unbelievably difficult.
Datadome reminds me of Cloudflare in their early days, when they were breaking UX for many users with Google reCAPTCHA.
I'm super interested in curating a dossier of these sites and experiences like yours, if only because we're genuinely interested in preventing issues like these and don't do the types of things that lead to these complaints, like IP-range profiling.
Disclosure: I work at friendly captcha.
I've also experienced Datadome I also doubt that these websites know that Datadome is messing up legit traffic to their sites
Have you tried a GDPR data request just because you can?
How would that help ?
they would need to produce all data they have about you. It may not lead to anything but you can mess up with them as much as they are messing up with you.
That's perfectly fine by me. I'll read those arguments and buy those things my agents can find on the open web; which is plenty.
Want to argue for a shining new world, but you're on a locked site? I guess we'll have to live with dystopia. Want to sell the greatest widget since the invention of sliced bread? Well, I never found it. Sorry, I bought inferior widgets from your competitor instead. I'm at a physical sales location, but I can't access your spec-sheets? Your competitor the next shelf over doesn't have that problem, I'll go buy theirs.
Exactly. Shut yourself off, you get shut off. I don’t give a damn you want me working the mouse on your site. Your site probably sucks ass and loaded with ads and popup offers and cookie banners and login hoops other bullshit I can’t stand that wasted my time. Oh check my email for a one time code becuase you don’t like passkeys.
I hope the irony isn't lost on these companies. Meta is one of the most aggressive companies against any sort of automation, perhaps after LinkedIn.
I'm trying to build a personal social media client to stop the platforms from shoving their feeds down my throat and to consume only the content I care about, so this hits home.
(I just got blocked from LinkedIn for letting an agent set all of the 77 email notification toggles off, which in hindsight was obvious but I wasn't going to do that manually since every toggle has their own separate page, it's a lot of clicks.)
Pro-tip: it costs very, very few tokens to have an agent (harness) sleep a random amount of hours between each toggle. Just tell it to. It's not burning tokens on `sleep`, just to call it.
It might not hit input cache at that rate but better than a block or ban. Keep the context window small.
Everyone is for their own automation and against their competitor's automation. It's just how business works.
> I hope the irony isn't lost on these companies.
"Companies" are not capable of feeling irony. They absolutely will demand the unfettered right to scrape your stuff while demanding that your scrapers be outlawed at the same time, and if you go "hey, that's hypocritical of you", you won't get any response, not even a half-hearted rebuttal or uncaring shrug. Taking about "hypocrisy" or "philosophical consistency" with regard to corporate behavior is almost a category error, these aren't concepts which even exist in that space.
Companies don't want it to be easy to cross shop their competition, the whole premise of Amazon is that it's so easy and simple you don't even check the competition. AI agents are leveling the playing field and helping customers find the best deals, so of course they're being banned
Everyone focuses on Amazon here, but there is an entire industry of e-commerce tools and products and platforms and services designed around optimizing e-commerce sites for sales and other uses.
Agents invalidate a lot of that and so it is an existential threat to many of them. I expect the near future will result in a lot of different results tailored to agents that may actually result in higher priced products for agents browsing versus humans.
Current AI agents overindex on price. The price has to be reasonable but I often rank fast shipping and easy returns higher than price.
Recently bought something online from Home Depot and it arrived damaged. When I asked them for help they said I had to drive it back to the store myself.
If I have to drive to the store to get it, I wouldn’t have shopped with them! It feels like every time I don’t order with Amazon the stores remind me why I ought to.
True, and agents can also present a comparison table of options including: total price, shipping time, return convenience, etc.
The last thing I want from an agent is a table of options.
Agents are likely the slow death of Amazon. Muse could decide to sell anything and group buy their way into the everything store.
Amazon's strength lies in their logistics/3PL/last-mile systems. They invested heavily into this very early on into their business and it's been paying off ever since.
Walmart is the only other company that can compete at that scale given their very mature ground distribution network and even then Amazon has them beat good on air cargo (Amazon has a virtual cargo airline operated by Air Transport/Atlas Air, both of whom they have some ownership over) and inventory management (they hired some very very smart folks to help them build inventory forecasting systems)
As long as Amazon remains the only company that you can buy almost anything and get it in two or three days almost anywhere on Earth, they won't need to worry about the grim reaper coming for them.
Agents cannot provide free 2-day delivery or easy returns
Even if they could, it will be a _long_ time before the average person uses an agent for personal tasks like this
It's really hard to build a walled garden just for human users, and not for agents. Have to pick a side.
It's not that hard. Make an iOS app and use the remote attestation features to verify the client. In theory Android has similar mechanisms available, not sure how secure they are but Apple's is pretty good.
Now whether you want to restrict your shop to only being accessed via a mobile app is a totally different proposition.
Or make users scan their passports to log in.
Your first sentence makes sense, but not following your second. Who needs to pick and what are the sides they are choosing between?
It's mostly because of ads though. Not cross shopping.
An easy solution is to sell a "web search" subscription add-on that gets distributed to the sites your agents utilize. The script will flip overnight and ASO (agent search optimization) will be born. It could be a form of micropayment system that the internet has desperately needed for over a decade now.
I think L402 would fix this.
https://docs.lightning.engineering/the-lightning-network/l40...
technical solution to a social problem
This kind of optimization is already a thing for at least a couple years.
It's called GEO
[flagged]
Without a convenience moat, commerce will get even more adversarial: membership pricing, surveillance pricing, loyalty perks, varied products to avoid price comparison (like mattresses and Costco).
Fortunately, if AI is really superhuman in capabilities it may still win the battle for the customer and keep prices competitive: aggressive price comparison, alternatives discovery, negotiation, group buying, etc.
try anything and everything except providing a better product.
How would the agent know it's a better product? All it has to go on are whatever bot-generated spam reviews and prompt injections that they can scrape.
In a fully agentic shopping experience, it's not obvious to me that providing a better product works to drive sales. Bargain-basement stores can make up a story about how their goods are lovingly handcrafted for decades of durability (adversarially tested against the most common agents, so they'll believe it) much more efficiently than I can honestly provide one.
This is just like all tech industry disruption.
First they forced their malware bot farms on us, now they want to rent them to us so they can turn the entire web into the dark web.
And eventually rug pull off the major retail sites and just redirect traffic to their own darker sites to scrape commissions, sell direct and keep the change.
I'm sure the teams thinking this up buy their lunches from a ghost kitchen run by illegals that's never had a health inspection,
located in the back of a laundromat with a phony business license and website that only accepts crypto.
I think people who rally against Amazon in this topic assume that the "savings" from much open price competition immediately come back to the consumer or the manufacturer. In my experience, with previous such tech improvement such as 'uber disrupting taxies', this has never been the case.
Both the consumer and the manufacturer/labor/driver/gig-worker always get the short end of the stick.... once AI-agents becomes indispensable part of life.
Uber is a single centralized platform that sets prices based on their secret algorithm to maximize profit, gives worse deals to drivers who look desperate and shows different amounts to two riders standing next to each other. It's as far from real competition as you can get.
It would be way different if it was an open network where drivers are allowed to set prices and reject rides without any penalty, and riders can send multiple offers at a time then take the best one. That would be a real competitive market where everyone is free to choose what they buy and sell.
In that scenario Uber would be one of many apps you can use that index the network, whitelist trusted drivers and handle payments for a small fee. And if you want to make a new app or do the whole process manually there's nothing to stop you. That's what shopping on the open web is like.
I am not disagreeing on any of those points. Just pointing it out that tech disruption only happens when someone else finds a way to steal that cut. It is never about the common good. That is why true open solutions and markets are hard to find.
The centralized platform in open-web shopping is Google, which is how you discover shops.
I would be surprised if you had enough data to be able to conclude such a lofty claim, especially accounting for increased availability of taxis in areas where there were none or few.
Either way, the benefits of price transparency are not imaginary, but they are also not the only factor. Things like insufficient buyers or sellers, government policy, etc are also in play.
All else being equal, a buyer being able to easily see seller A is more expensive than seller B for the same goods will result in lower prices for the buyer. But all else is rarely equal.
it looks like this is solving itself in the usual and productive way:
> Instead of trying to fight Amazon, the company has partnered with retailers like Walmart, Gap, Sephora, Best Buy, and others, allowing Muse to shop on their sites. At the same time, Shopify has also opened its network of storefronts to AI agents.
they're making deals with the sites so that the agents can work, the sites get compensated. I hope for this "AI can do what it wants if it pays" model to expand further. Cloudflare told us months ago we'd have an option to get paid by AI crawlers. I signed up to join but of course as a nobody I never got a reply.
That's how the crawler deals work (Reddit licensing its data, Cloudflare's pay-per-crawl). For shopping agents, Meta has described the money going the other way. Zuckerberg on the Sources podcast in September: the plan for Muse is "a very small cut of whatever the transaction is… It'll come from the businesses that they're working with." Shopify currently tells merchants there's no fee, so this is the plan, not the present.
So the stores aren't being paid to let the agent in. Meta expects them to pay it.
https://www.youtube.com/watch?v=Lx8lrn-cytc&t=1551s
This is just because agents don’t see ads right? Regardless, this is one of the advantages of software heterogeneity. The cost to detect rises substantially and the false positive rate will eat you alive. My agent buys me things. On Amazon too, yeah. OpenTable is hard to handle but I’m sure it’s fixable.
>This is just because agents don’t see ads right?
Put another way, because it undermines the revenue that is providing you the service. Working around it is not a sustainable solution.
I will always block ads no matter what. I don't care about what is sustained, there will be some way to get these things, and if not, we will get used to that too. Meanwhile I will not tolerate ads.
Indeed. One could reasonably argue that viewing ads on Amazon plausibly lowers the cost of goods served, just as connected TV targeted advertising allows very low cost TVs. I think that’s quite believable. Given that, I’m a free rider, somewhat like a software or media pirate.
Not outlandish. The prices are a big motivator and I wonder how much they’d have to charge me to compensate. My agent probably sees the sponsored results before it sees the other results but that is probably considered insufficient compensation.
I think I’ll keep doing what I’m doing but primarily because if the world is forced to charge higher prices to compensate for the lack of advertising I think that isn’t an outcome I particularly am upset by. It passes a Categorical Imperative Test, if you will.
I've never updated on this: The web was better when there were little or no ad-revenue based services. (and I'm ok with actually paying for services that add value)
It's the latest iteration of "a little X is good, a lot of X is bad".
Email was awesome when it first became widespread. I could cold-email random professors at other universities (I was in school at the time) and get meaningful responses back. Then lots of people (and companies) started cold-emailing, and it stopped working.
Social media is broken if there's nobody there, is great with a medium-sized community, and is broken if everyone is there.
Ad-supported websites can offer all kinds of valuable free services. Up until either (1) they become so ad-infested that they're no longer useful, or (2) all the users block ads.
There was a wave of websites providing APIs for programmatic access a while back. You could script together whatever flow you wanted. If no API was offered, you could scrape the HTML. That triggered a wave of alternate interfaces to sites, skipping their ads, skipping their engagement funnels, separating the value from the cost. So everything got locked up behind captchas and accounts and API keys with limited access.
This outcome was always inevitable. Humans don't understand social contracts, or they do and both sides abuse the hell out of it. We feel entitled to have just the loss leaders without ever buying anything else. When running businesses, we feel entitled to keep adding more and more adds and generally enshittifying the experience. People can take it or leave it, after all.
Agents don't change the underlying structure at all. They just magnify and accelerate parts of the system, and will therefore magnify and accelerate the reactive and defensive responses.
A social contract towards a profit-maximizing corporation is not real. They are amoral entities with only one goal. The real implicit contract between you and a business is simple: they will do everything they can to make you give them money, and you will do everything you can to not give them money, or else you're a sucker. "Everything they can do" includes convincing you there's a social contract involved.
The Internet is the embodiment of the tragedy of the commons, at scale.
[flagged]
The whole web is based on selling people's attention, providing free services as "payment" to people. The whole value proposition of AI is to give the services without giving people's attention?
Not "give" take.
Look at spotify and their royalties. Theres already such little value in human based internet that agents will race to the bottom.
The only internet content will be worse than adware. Itll just be agent poison meant to take your money.
Any other viewpoint seems to ignore the already shattered shared internet.
The openweb will go the way of usenet. Bots talking to bots, fetching information for a higher tool.
I have yet to meet a website I can't crawl with Chrome MCP. You just need to go slow and do extra actions that look human. When there's a captcha, AI notifies me to solve it, I do, it continues. All this is for personal use.
For this reason I created an overview with sites that don't block bots but let them shop without getting blocked.
https://shopsforbots.com/
my prediction: ai agents will have their day, but not on this version of the web. websites will have a separate xml/json/yaml version that agents can peruse without all the ads/js/etc
Isn't that just API? We had them. Reddit/Zillow/Wells Fargo/etc restricted theirs due to business reasons, not technical ones.
My agents can just take that feed and render it into a custom front end just for me too. Why would I browse the adversarial, bloated, ad infested version?
Why would that happen if the website owners are explicitly trying to prevent access? This isn't a technical issue, the agents are perfectly capable of working with the existing websites regardless of ads and javascript. The semantic web and machine readable representations was a 90s thing, agents don't need that really.
no you are def right the agents are def capable and that is the problem, so capable we are all doing this, we are all making our own scrapers, as are big corps/small corps and it is adding up so now we are seeing managed challenges everywhere and webmasters/sysadmins going to great lengths to block any bots.
the js/yaml version would require a license key that connects to a business arrangement between the website and the owner of the crawler. the plain HTML version would adversarially block them (or even identify them, where they could be sued...once we have legal precedents for this stuff).
And finally, we will have the future SOAP and XML schemas promised us all those years ago!
Websites cost money. Websites won't cater to bots the same way they don't cater to the scrapers of old. Some bot companies may be charged extra to integrate their bots in some websites, though!
I agree. I’ve been saying for years that marketing teams need to switch to include marketing to LLMs.
I have (currently free but maybe paid in the future) accounts for side projects with businesses that Claude chose for me. (I reviewed the suggestion, but didn’t see a problem or reason to change it.) Why did Claude choose those providers? No idea, but if it’s through subtle marketing to Claude, then well done to them.
Without the ads? The pessimist in me thinks with all the ads, prompt injections, and long-form content marketing included. Already you can see Google's AI results rehashing marketing slop in the answers about how amazing one product or another is.
Wasn't there some restaurant review site that gets accused of doing pay-to-play hiding of negative reviews and promoting positive ones that calls up businesses and basically extorts them into their services? They have all the incentive in the world now to manipulate the results shown to AI agents. You won't even see the manipulation directly yourself when there's an AI sitting between you and the source acting like an obfuscation layer.
hm. for me, AI agents effectively already have their day unless there is a retrenchment. i essentially do not interact with the web (outside of a few social media sites) anymore
>i essentially do not interact with the web
Grim. No point in writing blog posts or publishing anything for free if that's the future.
There IS actually. In my case I'll look at a site if an agent says it's good!
if the reason to do so was the ad revenue, then i definitely agree. but in terms of me happening on to your ideas, now is perhaps actually better than any time in the past. i also do pay for writing à la substack
Yeah the web is done, youtube is probably the only place worth publishing to now.
AI agents have already colonised the human internet. No one is gonna make them switch over. It’s the humans that are gonna be displaced.
This is why I think Cloudflare's new service to offer micropayments to agents is going to win and convert the ad ridden web to the ad-free web
Sheesh, that's hell of a plan. Insert yourself as the middleman for almost all internet traffic, push for a payment standard and then take a measly 1% as a fee. Boom, infinite money.
We should create a human knowledge base for AI to train on that is completly open so everyone had access. Not just big corporations. Then adopt Ethical Acceleration. https://youtu.be/nV3tBWnZ4wg?si=UX2lKXXPZr01FXYi
If only it would be possible to "close your doors to AI agents". It's not I'm afraid. Where do you draw the line? What browser is allowed?
For example, I now have to `pacman -S chromium` to buy tickets from Deutche Bahn. They accuse me of being a bot when I use firefox on linux. It's a known problem, and I think stuff like this will only get worse the longer this cat and mouse goes on.
http://archive.today/kqpFJ
In the wake will be the rise of private websites. No social media, no public profiles. Your own private circles.
Public only as accessible by government issued identity.
We host a website in Canada (physical server hosting VMs in a Canadian DC) and with the ongoing trade-war we do whatever we can to avoid anything US-based tech such as Cloudflare. So far we use Anubis to protect our services from scraping and it's been a decent experience.
Weird time to do it IMO, right on the brink of agents potentially becoming the main entity closing purchases for many. If agents succeed at becoming e-commerce middlemen for the masses, I think these decisions will age poorly and there will be some walking back.
Entire networks. GPT dot became useless as it got systematically blocked by Cloudflare, Google cloud, vercel, etc. At the same time OpenAI blocked solutions like tailscale and others effectively killing the utility of browser control by dot.
It’s still okay at email.
This is opportunity.
And the other side: If my agent cannot see your site, it does not exist for me
what are the ways to do it? considering that now people are looking for ASO (AI Search Optimization), won't this effect them in the long run?
[flagged]
Like Reddit is shutting down the RSS feeds :(
I mean there are multiple reasons not just the AI agents but that's also a part of it
Finally, we are at the dawn of the next wave of disruption.
Web 2.0 went from serving users to profit maximization. Amazon became unusable. Streaming services fractured and fragmented, with content locked behind numerous paywalls. Ads permeate every surface.
This is not to say the agentic era will end differently. But at the onset of this era, and for the foreseeable future, customer adoption and trust will decide winners and losers.
I don't want these "agents", aka private tools of greedy mega-corporations, to ruin the world wide web.
I constantly get pestered by "are you a bot" queries.
They ruined the world wide web here. I did not have that in the 1990s or early 2000s.
Instead of closing doors to AI Agents, it is time to start issuing `402 Payment Required` statuses and have the agents pay per use.
I mean, yeah; agents and AI training data spiders basically operate like DDoS swarms, and it's not like their harnesses implement client-side rate limiting or anything like that. WAFs were designed to block/slow exactly this kind of traffic, so make sense that businesses will use them as designed.
China doesn't care. They'll scrape you anyway.
The world doesn't care. They've got China firewalled away anyway.
It is good that agents can't possibly use puppeteer.
When people were uploading music videos on YouTube, media companies fought to the death to prevent that. Then suddenly they 180 and started promoting YouTube videos and begging for views.
This will go the same way, in 2 years 99% of Internet traffic will be AI agents. You close the doors, you stop existing.
Recently I've asked an agent to find me a laptop with particular requirements. You do not let the agent search your site, you will not get the sale.
https://archive.ph/kqpFJ
Right now Claude/Codex are pretty good at following a websites TOS and refusing to solve captchas.
I expect that will no longer be the case in the next year.
This annoys me. I can understand stopping mass scraping by big labs, but if I tell my personal instance to try and find the best price for a 40" monitor, it should be able to get details from any site. It's acting as my proxy.
Not only this, but the AI will start picking up the scam websites that allow it to search it.
Most content has been behind a paywall / login wall for years
Solvable problems (just login or pay). This is more than just blocking anonymous scrapers. Sites are blocking access to agents even when they have a proper "ticket" for access.
Sites are blocking access to agents even when they have a proper "ticket" for access.
(Shrug) Well, they must not want to sell anything to me.
This is not a battle that site owners want to fight. They may think it is, but they're horrifically wrong about that.
It's insane to me that people don't see that agents are going to be how you interact with your computer going forward, almost exclusively. The people who make things hard for agents are going to get eviscerated by the people who don't just make it easy for agents to do things, but market primarily to agents rather than humans.
> It's insane to me that people don't see that agents are going to be how you interact with your computer going forward, almost exclusively
You are in a bubble. Normal people (i.e. 99% of people) will not use these agentic tools:
* Some, probably most, won't use them because they don't trust them
* Agentic tool providers can't possibly supply everyone with their own free agentic tooling
* The vast majority of people will not pay providers what it will cost to run these agentic tools (or even anything)
* The vast majority of people cannot afford to buy their own hardware to run their own agentic tools in-house
* Agentic tooling is mostly useless for most people (who cares that it can check you into flights, most people don't have that kind of problem)
If that's true, why bother blocking the tools? Either they are useful in saving money (and more important, hassle), or they're not. If they're not useful, the problem simply won't materialize.
Because not blocking these tools causes additional operational costs.
For example, higher return rate for online shopping (because the tools didn't buy the correct thing, or maybe they bought something that they shouldn't have).
Or, the tool ignores the adverts on an ad-supported website, intended for humans, basically just adding costs for no benefit to the site.
Or, the tool wastes customer services' time on something that ultimately the agent's operator didn't even want in the first place.
They're just off the top of my head.
It's a mistake to think the sites have a choice. I'm hardly going to trust an AI agent to shop on my behalf if it isn't smart enough to operate the browser on my behalf in a way that's indistinguishable from a human. They're almost good enough to do that now; it won't be long.
So the end result will be that the sites spend money on bandwidth to shove ads and dark patterns at me, which I will never, ever see.
You'd think they'd understand who's going to end up with the power to say "No" here, but apparently not.
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
[flagged]
[dead]
[flagged]
[flagged]