and a handful of other ASNs on almost all of my properties. It would be one thing if it was simple scraping, but it's often genuine agentic abuse (just spamming for vulnerabilities* or scraping millions of pages with very little valuable or dynamic content). These clouds must get the abuse happening under their own roof under control.
I banned Alibaba's Cloud two years ago and it still hits one of my domains 1 million times per day, despite every route 4xx-ing during that time.
Even Apple has aggressively ramped up its scraping in the past few months, but thankfully they do respect robots.txt.
It's frustrating because some of my real users route through these clouds via their work VPNs. Also, there are a lot of less-than-reputable AI scrapers using global bot-nets via residential IPs to scrape, making it even harder to detect and ban them without affecting real people.
* like leaked .env routes, brute forcing admin panels, and so on
Bots and spiders hang onto old URLs longer than I hold onto a grudge.
We had some SSR that followed up with JavaScript listeners to do interactive updates instead of loading a whole new page. But the team responsible didn’t add nofollow and some bots ignore it anyway. Some significant fraction of our cluster size was answering those requests. But one deployment someone broke the fuck out of those links and it showed up immediately in our 4xx error statistics and we rolled the change back within a week.
Three years later I was still seeing those busted URLs being submitted.
Then the fix for the nofollow got implemented and declared done but the flag never got made permanent. We were pretty peeved when someone learned that in some offhand comment in a conversation between teams.
We were already identifying “good” and “bad” bots at the border and telling them to fuck off, so all of this was coming from our supposed friends. And that’s when I realized that a company owning a search engine and a cloud service provider is a protection racket. Google cloud is selling you servers in order to be talked to by Google bots.
This has been going on for FAR longer than LLMs have existed. We call it "internet background radiation", because it's everywhere; it usually shows up within minutes of a host coming online. Or just about immediately after your first LetsEncrypt certificate gets issued.
Pretty much every infosec company of non-trivial size runs one set. The old venerable services are Shodan and Censys. Then there are a various security "researchers" who like to point their crapware vulnerability "scanner" at a subnet or ASN and let it rip.
And then there are of course all the actual botnets and low rent access broker wannabes trying to pop shells where ever they can, so they can resell that access.
Application-level banning seems very inefficient, these domains should be banned at the network (IP) level. Just drop any incoming packets from these domains/IPs, send absolutely nothing back that even confirms that you exist.
From a domain registry operators point of view Cloudflare are the worst of the bunch by orders of magnitude.
Google DNS are busy but its a hand full of IPs and reasonable. Cloudflare is non-stop never ending traffic. I don't know WTF they are trying to get. I've also seen TCP queries from those same Cloudflare IPs so this is not the result of a DDOS attack on them.
This - but unironically. It has been striking how fast the public Web has descended into bot wars. I think something new will probably start to take over relatively soon.
People who used to contribute valuable content have pulled it because the bots were scraping it and reusing their work for free. Of course that was always technically possible but it used to be considered bad manners and not enough people did it to break the whole model. That's evidently no longer the case.
People who want to access real content are now interrogated with the assumption that they're probably a bot before they can even see a basic web page. Those interrogation systems are themselves are now running through a very small pool of providers with far too much influence and control to be healthy.
One of the most valuable means of communication and shared knowledge we have ever invented is being effectively destroyed before our eyes. I don't know what the solution will be - though I fear that the idea of privacy and anonymity online will suffer serious collateral damage - but I suspect that some new information network only available to people willing to identify themselves as a specific real human being in some relatively robust way will be a big part of it just as many sites are now gating their content behind user accounts.
I wonder whether the latter might be sufficient: have your terms state that user accounts are only available for real people and by using any automated system to attempt to create one the operator agrees to pay the site operator some ludicrous amount of money in recognition of the damage to their operating model that is posed by deceptively allowing automated access. And then make sure that such terms actually stand up in court - by legislating to enable them if necessary - in enough of the world to kill the AI scraper/bot model.
Can you do something with fail2ban? I.e. after a small number of requests resulting in 401 or 404, or a series of requests that's too fast to be reasonably from a human, ban that address for 48 hours.
If a legitimate request later comes from that address, it will probably get through.
But not sure how well iptables handles checking every connection against a list of possibly hundreds of thousands of addresses.
All those boomers out there with free IPTV just don't understand why it's free, my in-laws are guilty, but hey, they can glaze over and mindlessly melt like it's the 80s in different timezones and across continents
Does that old slowloris thing work but in reverse? Up your connection limits and keep N alibaba ones open forever, drip feeding them a small stream of data until they leak or timeout.
Most big services will just take the junk traffic. They're never going to block whole cloud providers.
Blocking cloud providers is strictly for us dirty self hosters who are sick of our logs becoming unreadable and our visitor metrics useless from all this malicious traffic.
Does the money add up for that to be a deterrent? Are those using them for general VPN access paying enough that they'd care much, and if so is that enough compared to what they'd lose by kicking off the users abusing external sites (they can't all be using just free plans, and anyway tests to kick them could have enough false positives to lose them good paying customers as well as bad users). In any case is the amount of traffic enough for them to care to spend the effort tracking it to identify the bad users?
We spent a decade wasting cryptocurrency compute and still didn't come up with the decades old promise of a partial payment plan for web use. Seems like it'd make a lot of sense now that ad revenue is basically garbage and repetitive traffic is clearly just leeches and parasites that should pay for freeloading.
Someone hack a payment gateway into every webserver and MCP and let's get the bloodsuckers to pay for the internet.
> like leaked .env routes, brute forcing admin panels, and so on
That sort of thing was very common long before agent based abuse scanners. Volume would wax and wane but never went away.
Though other scans for vulnerabilities have shot up in frequency now everyone script kiddie and their flee-bitten mutt can ask an AI to do it for them.
However, they could just use residential proxy botnets.
I believe the future is static resources with dynamic apps being only offered to people who can authenticate. And even then I’d recommend grandfathered accounts (eg telegram users that had an account for at least a year).
I’ve had to block similar on clients sites. Their search pages would start to get hammered with obvious agentic search spam. Searching for thing not even related to the site in all combinations of the queries. Some of the user queries contained potentially PII.
If there's a know CVE in a version of cPanel ir wordpress for example, those platforms broadcast what version they're on, so the matching cve being known is not illegal. You're just correlating known vulnerable versions without doing any probing. Pen testing however, a bit more involved in pulling info out without permission is illegal.
Can you enforce it? Would you even know who to go after?
Traditionally cybercrimes have been prosecuted by the victim going to their local PD (or the FBI) with the IP address of the offender, and then law enforcement subpoenas the ISP associated with that IP address to figure out the customer associated with that account, and then they have a case for prosecution. It all works as long as cybercriminals are less numerous than law enforcement, fall in the same jurisdiction (or in jurisdictions that have reciprocal extradition treaties), and have a real identity that can be tied to their online activities.
Now, many law enforcement agencies are tied up prosecuting political opponents of their bosses. To the extent that they have bandwidth to investigate and prosecute ordinary crimes, they're going to prioritize the ones that target a.) the government that backs them (of which there are many - you see another headline about the government being hacked every week or so on HN) b.) major corporations that have sophisticated IT systems that can aid prosecution or c.) headlines that will get their boss a PR boost and themselves a promotion. Your personal website does not register among those priorities. You probably can't even prove damages from the intrusion itself. The perpetrator might be a Russian bot that the U.S. doesn't have jurisdiction over, or a Canadian teen that the U.S. previously might've been able to prosecute with reciprocal judicial treaties but now has no leverage over because we've blown up our relationship with Canada. It may be legally a crime, but economically it's not worth prosecution.
Traditionally security researchers had to be very careful because they were doing everything above-board, with their real identity attached to their actions, and they made a very attractive (and PR-worthy) target. Somebody who's port-scanning from Russia through a VPN or residential proxy network is a.) anonymous b.) untraceable c.) out of U.S. jurisdiction and d.) not a headline-worthy target anyway.
A couple years ago I predicted here that a lot of things will break as the U.S. transitions from a high-trust to a low-trust society. This is one of them. Much of our justice system works well when criminals are rare and utterly falls apart when they're ubiquitous.
Last year, I wrote a post about how I use zipbombs to deal with bot traffic [0]. It went mega viral over the web, and I regret it. My server was completely overwhelmed with bot traffic. After months of dealing with it, I had to find a better strategy then keep my mouth shut.
Last year, I read your post and implemented it. I had no way to tell whether there is any benefit to it, so I discontinued it. Serving the data, even though it is compressed and the compression is precomputed, was still a waste of bandwidth. It could hit the browsers of legit user (false positive).
The main problem on my site is that bots hit the CGIT repo, where visiting files triggers syntax highlighting scripts.
I put in a skill-testing-question type system where you get a maximally long-lived access cookie if you answer right (so legit users are bothered as infrequently as possible).
CGIT doesn't support client-side syntax highlighting, as far as I know. Anything could be hacked, but out of the box, it has a system consisting of dispatching a script.
I have custom syntax highlighting. For instance, I have a home-grown script which takes a man page file (identified by a suffix like .1) and renders it readably, so it looks like a man page on your terminal, rather than nroff source code. (The only unsolved problem is that CGIT renders a column of the original line numbers down the left, which exceeds the length of the rendered page.) I also have accurate syntax highlighting for a home-grown programming language. That syntax highlighting exists only for Vim, which has a superb syntax highlighting engine. So in fact Vim is launched, via expect, to do some HTML rendering! There is no way that is getting translated into client-side.
Pre-rendering is not really an option over numerous git repos, that have numerous files in them, with the added dimension of history. What CGIT has instead is tunable caching.
Bots which aggressively scrape the history are hostile to the caching scheme, however. It works best for human visitors which mostly look at the head of the master branch in each repo, and poke around just a little bit in the history or look at other branches.
It worked fine... on a small scale But it's one thing to swat flies versus fighting swarm of yellow jackets. Serving a 1MB file a million times a day on a small digitalocean droplet is not feasible. Let's just say it didn't scale. And AI bots usually come with beefier machines, and recover faster after a bomb.
It's the content of the page, the raw html, that is replaced with a zipbomb. Example: https://example.com checks if you are a malicious bot, depending on the result, it will send you gzip html or gzipped bomb. So that can't be swapped or redirected to a CDN. The bots would simply ignore the redirect.
I have a small website with a contact form that gets a lot of spam. I found a very effective way of filtering spam, but it only works because it's not mainstream, so I can't ever share it :(
I posted a link to an online game I made in the comments here once, and immediately got all kinds of good and bad traffic.
Some people liked my work, found me on LinkedIn and sent a connection with a positive message. Other people tried every which way to SQL inject or attack the site, posted slurs in every available input, etc.
Someone copied the site and hosted their own version with a couple minor changes. I only found that one because they had left one of the links in, which gave me referring traffic from their copy.
All that from one HN comment, very much pre-AI too, not to mention the standard daily slew of requests probing for what I assume are known vulnerabilities in Wordpress or PHP.
The internet is a floodgate of different actors, many of which are automated, and a link posted somewhere like HN opens the valve especially wide.
It's too late to comment on your older posts, but I'd just like to say: it's nice to see someone making oldschool web browsers work properly on the modern web.
I was going to comment on that too. Talk about burying the lede! The idea of using an LLM to generate a static program for a specific web site is brilliant. It reminds me of the second Futamura projection.
I rolled my own version of Anubis for reasons*, and > 80% of all traffic is bots that fail the challenge, even when white listing legit crawlers.
* I wanted an elixir version so I could run it alongside my existing apps instead of having it sit as a proxy in front of them. This way my existing apps can validate if the caller is already whitelisted and if not redirect to the challenger https://sphinx.8by3.com which sends you back to a site if you pass eg. https://tuhat.net
If you are rolling your own it probably doesn't need to be as clever as Anubis is trying to be. Simple something relatively CPU heavy in JS will do the trick, bonus points if you can make it memory-heavy too. Most bots will give up running something CPU heavy currently (though they will run a lot of JS stuff as much of the web is inaccessible without) and they aren't going to optimise your challenge in WASM or similar because it isn't worth it for the small number of sites it protects (Anubis' challenges on the other hand may well be worth optimising for that way because of their wider use, hence that project is stuck in the cleverness war of attrition and definitely needs to move toward memory-hard problems and something more efficient than JS sooner rather than later).
My main reason for planning to implement something like that myself instead of something off the shelf is two-fold:
1. The points mentioned above
2. I can be a dick and make some subsets do harder work (longer challenges, adding time delays that wouldn't phase a bot) - “hi there North American IP, I know how much you like extra hassle hence the love of effectively paying for tarriffs, here, have a little extra work on me!”
The challenge isn't terribly clever, just some basic crypto maths, same as Anubis. There are a very low number of AI bots that pass it based on my logs, like 0.01% low and I suspect they are interactive agents that bother to run JS as crawlers typically don't. Assuming these are actually run on an end users computers I don't think any amount of cpu/memory weight would be a real blocker. One thing I've seen that is interesting'ish is bots failing as they answer the challenge from a different IP than it was requested from, and the source IP is validated as part of the response so they fail.
> that bother to run JS as crawlers typically don't
I'd be surprised if many don't run basic JS, given how many sites require it for basic content these days (load base site, then make a fetch request for the actual content), unless there are less such sites than I think.
> One thing I've seen that is interesting'ish is bots failing as they answer the challenge from a different IP than it was requested from, and the source IP is validated as part of the response so they fail.
From what little I've looked into the details, it seems that scrapers (both old and new) using a scattering arrangement (built in or via an outgoing proxy) seem to be split 75/25 between those that completely scatter (following requests will almost certainly come from a different address) and those that stay on the same address for at least the first few requests (which would cover login, bot challenge, or such) then move around to avoid requests-per-source-address based resource management measures. Only really simple ones (someone running a recursive wget without proxy) stick to the same source address all the time now.
Unfortunately you can't be more strict about changing source addresses because of how many false positives you will see. CGNAT arrangements often move people between addresses fairly regularly, and even without that there is the common case of people moving away from a WiFi AP and automatically flicking to their mobile network (or another nearby AP).
> “hi there North American IP, I know how much you like extra hassle hence the love of effectively paying for tarriffs, here, have a little extra work on me!”
I'm assuming you're outside the US based on having this idea.
You might as well just say "I'm mad at your country for the tariffs, so here's an extra wait." From the fact you feel the urge to spite people in a tariffing country like this, it's clear the tariffs have had some negative effect on you, which goes against your narrative that they're "effectively paid for" by those in the tariffing country who voted for it.
You have a some incorrect assumptions there. Yes, I'm outside the US, though I can't say I've experienced any issues at all that can be attributed to the continuing little tariff spat. I have heard of US citizens either complaining about cost raising because of them, or if not complaining then acknowledging them as something they consider a necessary pain. There are a fair few things I'm mad about, some of which do stem from the current administration over there, but this certainly isn't one of them. In fact I'm currently amused by it and the urge to mildly irritate those state side is in order to extend that amusement a touch.
I see, you're supposedly just amused by annoying people... that certainly makes you look better, I can see now why I was downvoted. /s
If you think your nagware is going to get back at US citizens for completely unrelated policies you're mad about, then maybe you should list those policies in your message instead of bringing up tariffs? That seems like it would be more productive to your goals, and less likely to make it look like you're salty and/or making a faulty argument regarding tariffs.
> I see, you're supposedly just amused by annoying people...
Some people, yes. I'm not claiming to be a good person, especially not that this makes me one, in fact stated “I can be a dick and…”. Some of my interactions on social media are similar - I do it to vent a little, for a little dopamine hit & mental health boost, or just because I need a short escape from work/chores/other for a few minutes and the get-some-fresh-air, or play-with-the-cats options are not available at the time. I call it my “breakfasts with the Percy's”.
For the record I, except for very rare occasions, don't downvote: I either comment or move on. Most likely the people that have downvoted have done so not because they disagree with you (maybe they do, maybe they don't) but because that post was the point we stepped too far off-topic, so perhaps try not to take that personally.
> more productive to your goals
Why do I need to be working to a world changing goal beyond a bit of “isn't the regime over there a bunch of great steaming wazzocks?” banter that might amuse some people I like, and maybe irritate some who I disagree with¹? Some things are just little in-jokes, not everything has to be for grand effect in some more meaningful agenda.
--------
[1] though they'll likely never see it, this is stuff I only put online for me and a few family/friends, including a couple state-side. Even the scraper bots haven't found it yet (the look-for-known-vulnerable-software scanners pass by regularly, so that is likely only a matter of time) so I doubt a mass of random human visitors are ever going to flow in.
I want mini games! "Score 5 points" style. Although, it would surely fail ADA compliance. I don't see how any of this could be practically effective, respecting ADA.
In server logs you can distinguish the type of LLM's traffic. OAI-SearchBot/GPTBot/ClaudeBot is just indexing but ChatGPT-User/Claude-User etc. may be valuable traffic generated by user's conversation with LLM's. So it's worth choosing which ones you block, instead of blocking all of them.
What's the value for the website maker in these? There's user value to it (rather than just value for Anthropic/OpenAI) but the whole point of these bots is to make sure nobody ever needs to visit your website.
Tons of people click from ChatGPT! It's now the top referrer on most of my websites, and seems to be on path to overcoming Google soon (a ton of traffic from Google has vanished since the AI Overview rollout...).
I realized its much faster to do the "google thing" just with chatgpt. Its filters all the sh!t for you. Searching the net frustrates me most of the times this days. Search engines are software for an older generation of the www.
In my case this is the main source of registered users. If a user is engaged in a discussion with chatbot and gets the recommendation from it his engagement is higher. For example, in my system, a user has registered and within one hour used the system 7 times and stayed active while for registrations from Google I can observe zero-activity accounts.
Strong agree, a good way forward is letting user agent once again truly mean an agent on behalf of the user. And speaking anecdotally, I've seen different traffic "shapes" under the same UA, which leads me to suspect that some actors might be disguising traffic as more mainstream labs.
I’ve found that Claude is really helpful for finding patterns in my web logs and identifying bad traffic patterns. Meta was ignoring robots.txt on a wiki that I run and generating so much traffic that it was effectively DOSing the site until I blocked their crawler.
What’s been really handy is being able to identify “stealth” bot traffic by the request patterns (e.g., not requesting CSS/JS/favicon when loading the page).
I got $10K GCP bill one month because of ClaudeBot. Its likely some users were trying to replicate our software by using Claude code + chrome plugin. If I was spending $10K on traffic, I can only imagine the money cloners must be spending on tokens.
Well, potentially very little! It could be subscription usage, or just really inefficient code authored by an agent. It probably costs LESS in tokens to write a poor quality cloner that breaks upstream things.
To solve this bot traffic issue first I check website Real traffic by using SerpSpur Bot Traffic Detector (TAI™ Checker). I can use this tool past 3 months and my client website is ok from fake traffic. Thanks!
Basic auth worked for me. Locked down everything but the home page. Put the credentials on the home page. No more shipping my assets to nowhere hundreds of times a day. Of course, I had _zero_ traffic other than bots...nowadays I have given up putting my crap out there for no one to see.
> I won't name names, but I looked them up, and they describe themselves as a service that listens for changes to a website, and then summarizes them into an AI slop news letter. And by 'listens to changes' I mean query our website many many times per-minute. Blocked.
That sort of behavior would justify a name and shame imo
Inconsiderate shitty services don’t deserve the politeness of protecting their name
I've been posting about this recently but I continue to be bad at either timing my submissions or titling them. I struggle with YouTube thumbnails too so I think it might be me. But anyway:
I experienced the same thing after submitting articles from my small photography website. I did debate using some of the off-the-shelf solutions like Cloudflare, Anubis, or iocaine, but I ended up building my own L7 proxy that utterly fucks the bots up: https://robertmay.photography/journal/bwbach-my-guardian-gob.... I will make it generally available at some point once I get it optimised how I'd like, but currently I'm able to trap about 3000 concurrent crawlers in my tarpit of slow requests on two small Hetzner VMs. And that 3k is really just the limits of the networks hitting my servers right now, I think it'll go significantly higher than that.
I haven't really seen anyone else combat them this way, possibly because many of the standard proxies and servers are set up for fast requests, not serving them as slowly as possible. Deliberately serving page loads for as long as the requester will accept packets turns out to be a really funny way to stop the botnets - when I just rejected the requests they in turn hammered my server even harder. By tanking the request volume with a ridiculous BEAM-based proxy you can reduce their request speed to a crawl and I'm fairly certain a couple of them have blacklisted my server now.
I like the spirit of this approach; make them suffer! But I'm surprised the bots wouldn't have reasonable timeouts to counteract. How long do they typically wait on these requests?
EDIT: sorry, that was a dumb question before reading your post. You actually learn the timeout they tolerate. Nice!
I should probably put together some actual stats at some point, but from what I see in the live logs you'd be shocked - some of them will happily sit there for over 5 minutes being drip-fed packets!
Looking at you, Meta.
I do still get some requests going up to 15 minutes which is quite ridiculous, though those don't seem to be super common (maybe they're learning).
I think the shortest timeouts I've seen are between 2-5 seconds, but they're surprisingly infrequent and actually that's still a massive inconvenience for them. I did at one point just drop any request I couldn't keep open for more than 5 seconds but it actually caused more server load. I'm essentially inverting the server load by making slow requests more efficient, which is counter to everything I ever learnt about performance for web serving :D
I think the average I see is probably somewhere between 15-30 seconds. I'm guessing many crawlers don't want to go too low because they'll be trying to fetch large files like images, or perhaps they know they'll be slowing servers down and want to ensure they get the response. Most likely, the majority of crawler authors aren't expecting some arsehole to exploit this and they've just left the setting as the default.
Ooh that’s fun! I’ll have to add some stats like this to mine too. Anthropic keeping connections alive for 2 hours is wild. I don’t think they fall into my trap as I believe they respect robots.txt, which is at least an improvement on Meta.
You shouldnt defend bad actors and abusers. Hiding them is tantamount to them attacking others indiscriminately. Cause youre not the first victim, and definitely not the last.
You see something, you say something, or you do something.
2. Setup a Geo-IP allow list using MaxMind's country DB (https://github.com/P3TERX/GeoLite.mmdb). I only allow visitors from countries I do business with.
I'm trying to understand the reasoning behind the entity that was sending presumably hundreds or thousands of requests per minute to the same page. It's some kind of AI news site, so it makes sense they want the freshest content, but it sounds like they were doing many requests per second, so that can't be the reason?
I run a few private MMO servers, we have a website with a support forum, in order to post, you need to register, log in, fill in account data, then create a thread, each of these pages is protected by Turnstile as well as a general bot check for the website, we frequently get automated spam from RMT (real-money trade) services for the OG game. Turnstile is maybe effective against simple bots that just try to call curl/requests/whatever but if you have bots that use a browser driver it's completely useless.
Turnstile is no longer effective. Anyone with a budget uses a real headless chrome instance and gets around it. You only need the spendy chrome instances to get the initial cookie, then you pass that to your workers and they scrape.
It’s really starting to suck. The worst part is these are coming from consumer vpn providers that we can’t just block (lots of real users paying real money on our service are vpn users) or they come from big residential isps. The whole residential relay thing is a hard adversary, since it’s spread across so many real users paying ips that you can’t whackamole it. I’d say 1% of residential ips are participating in a botnet (there’s prob some real numbers out there if I cared to look).
This is just going to push everyone harder into walled gardens. Web input biggest inbound but our content is our mote, we are going to have to turn off non logged in content viewing, which will have real business impacts.
I did this for alyph.ai, but was recently flooded with so many browser-using, Turnstile-defeating bots from Taiwan that I had to completely turn off sign ups. It's not fun anymore.
Not even third party login worked (Google, Linkedin, ...) to curb this.
They were logging in, scraping the client API for vulnerabilities/possibilities and then spammed erotic dog poems to Claude Fable until the $5 welcome credit got drained.
Since there were a couple upvotes, here's one I saved and Google-translated:
<story-record>
## Phase O. This round is a key advancement in the long-term, solidified daily storyline: the two enter Zhou Zhiyi's apartment for the first time under the guise of "non-purposeful" activities, and the soup-making incident becomes a new variable in their relationship. The task is to let this soup ferment naturally, without ending it or revealing anything.
## Phase [Reflections on Life]
1. Vibrant: Wu San and Wu Si appear as living beings with distinct personalities; Wu San is a drama queen, and Wu Si is slow to warm up; Zhou Zhiyi's apartment retains traces of his real life (a rhinoceros beetle insulated box, vinyl records, Pokémon, and a doghouse).
2. Cognitive Barrier: Shen Liran's "disbelief that he can make soup" and "wanting to kidnap a dog" are information revealed through her words and actions, which Zhou Zhiyi can perceive; however, he
... [some more gibberish] ...
Then Wu San saw Shen Liran.
This dog's opportunism was on full display. Abandoning her father, she turned her short legs and charged towards the backless dress, her front paws gripping the hem, head tilted back, a precisely calculated whimper escaping her throat. Zhou Zhiyi stood in the entryway changing his shoes, coldly observing the performance, thinking, "She's been fooled, fooled. I just fed her a treat this morning; she's a master of disguise."
Sure enough. He counted to two, and Shen Liran had already squatted down, oblivious to the hem of her coat dragging on the floor. She ran her hands into her long, brown fur coat, her voice sweet enough to make candied fruit: "Oh my, you're so soft to touch! Did you miss me? Even your dad isn't as good at being affectionate."
"Three seconds." Zhou Zhiyi hung their leashes back on the wall. "I said you wouldn't last three seconds, and you didn't even last three seconds. You squatted down before she pounced on you; that's a false start."
Interestingly, in the markdown, which was Chinese mostly, "Phase" and <story-record> were in English.
It's unfortunate that this also just leads to port scanning and huge accidental bills. This happened in the past to more than one acquaintnace of mine; go and market their vibe-coded/OS project on HN + twitter and then suddenly the vercel bill goes through the roof with bots scraping the website to bits, even with bot protection from CF. Really hard to get away from it these days, but hopefully over the next few years some new startup will raise $50M for "killing cloudflare" and will actually follow through!
Certificate transparency logs get scraped. However, they cannot scrape wildcards. Get a domain.example + *.domain.example cert and host your blog on blog.domain.example and you can keep the bots from finding you, at least for a little while.
No, it wouldn't. Just yesterday I added a domain.example + *.domain.example certificate, and today some bot comes to scan for subdomains: api, app, backup, beta, cdn, ci, mail etc. Blog isn't in the list, but just for this bot.
What confused me most about this is that the entire site is, apparently, a single page (there's no sitemap that I can use to verify this; there IS an RSS file which appears to confirm it) containing all posts. Links at the top (headed "Table of Contents") are all in-page links.
At first, I assumed this was a single blog post, especially with a "Table of Contents" at the top. I guess it's following some old-school blogging conventions, but AFAICR, even those sites retained individual pages for separate posts.
You can also link between Org documents and org-publish will export the documents with working web links, if you're lucky. That's how https://entropicthoughts.com works.
if you do decide to give it a try, and are coming from vim, i've had a good experience with https://github.com/emacsorphanage/god-mode as a way to kind of bridge the gap into modal editing
I have only had to block Lightpanda user-agent for super aggressive scraping from a multitude of source IPs. I did have to implement heavier rate limiting for some application paths to reduce performance impact from other agentic traffic. Rather than wholesale blocking agents I’d prefer they see my applications as useful resources and drive potential premium api upsells to agentic developers who want bigger rate limits
I posted a link to HN a few days ago that got literally zero upvotes and zero comments (Applying NASA's satellite-imaging trick to ancient museum objects https://news.ycombinator.com/item?id=50023128) and it was hit with insane bot traffic. HN was the only place I had posted it.
The output side is just as lopsided. Google alone was at ~3.2 quadrillion tokens/month in May: roughly 79T words a day at 0.75 words/token, against ~100T words/day humans produce.
It happened to me too. I got a big cloud egress bill because my game[1] is basically a mirror of thousands of Wikipedia articles. AI bots would smash all the pages so my cache hit rate was low.
[1] https://redactle.net (come at me Claude)
If you have an SSL certificate so such a domain can serve HTTPS traffic if needed, which it wouldn't surprise me if some parking services generate for you these days via an ACME provider, then your domain will be in the certificate transparency logs so can be found that way.
One possible way around this is to use a wildcard and not serve anything on the base domain. Unless the right subdomain is listed somewhere, they'll not get any scans. Maybe useful for projects that you have online for just yourself and a small closed circle.
That's a vague and borderline off-topic suggestion, imo. When the problem is too many malicious visitors hammering your server, the amount of time it takes the visitors' browser to run the JavaScript is not the issue; on the contrary, when dealing with this issue, you might as well do more rendering client-side to offload as much work from the server side as possible (hence the Anubis-esque defense of adding random JavaScript work that isn't even necessary to render the real page).
I have tried a ton of stuff (Cloudflare, Anubis, fail2ban...) to combat the bots lately, but they all have their downsides, and users complained too much, so I just gave up. You simply can't win this arms race.
So I have resorted to performance improvements on my server and serving every single request. Things are way smoother now.
I recently posted a link that got 2 boosts and 6 faves, but it was requested by about 300 fedi servers so they could generate a preview.
Which is fine! Because even though my photo gallery runs on a 15 year old laptop in the basement, I wrote the gallery software in a fast language (Rust) and so a trickle of requests maxing at 4 per second is not actually a problem.
People complaining about fedi hammering their servers are probably using PHP or Python.
Meanwhile if you publish on GitHub pages and aren't notable you will apparently get barely any traffic at all :( (Unless the bots are somehow just deciding not to follow my third-party hit counter links, anyway…)
Wow, your product is a smart idea! But never really thought that these things were caused bya service turning websites into AI newsletters. It's interesting to me that this can cause the website to be slowed down.
> somebody is trying to access -/wp-content/phpinfo.php
This is normal, I see hundreds if not thousands of these every day in my logs. You can use fail2ban to filter out some of the noise, but ultimately what you can do is limited.
One option to reduce it, though this is work to setup initially:
1. Use a catch-all subdomain
2. Reject email to addresses where a simple checksum does not match (valid addresses being something like asdfghij.klmn@catchall.domain.tld where the klmn is taken from hash of asdfghij and some secret, anything not matching that pattern gets rejected) [this means running your own mail server so you can do that on incoming mail, running a scan to pick out and remove errant messages regularly, or having a mail client where you can implement this as a plugin]
3. On your contact detail page, don't give out a straight email address but have the the page run some Anubis style JS before asking the server to respond with an address that works
4. Once an address is obviously being used in a manner you don't care to respond to, add it to an explicit block list.
For extra protection from addresses being passed around (intentionally or via hack & other leaks), ask for their email address first and use that as part of the hash, then you can reject mail to your catch-all that doesn't come from the original other party.
Whether all this is worth the effort of implementing, is of course up to you. It probably isn't… I've been considering something like it, but mainly as a fun little project rather than something that would be massively useful.
This happened to me. My site got shutdown for awhile because of the traffic after I posted an article on HN. I think they are trying to DDOS me (puts on Tin Foil Hat)
>So I took a look at some data. I added top-level logging for every endpoint (how many times was a request made to each url path), and I found that a majority of the requests did not even make an attempt to fetch news articles! For reference, my website is built as a single-page-application, and what this means is that a majority of the requests did not even run the javascript necessary to render the page (!).
So you got hammered by curl requests! But isn't that part of your delivery static anyway? Were you regenerating the same HTML over and over?
>many, many requests per minute?
How many was it?
Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time?
> So you got hammered by curl requests! But isn't that part of your delivery static anyway? Were you regenerating the same HTML over and over?
i dont think this was the cause of the slowdown, it was just sort of the thing that led to the discovery that this was mostly bots.
in general, because the purpose of the website is to allow users to generate news feeds based on custom criteria, it does do a bit of meaningful work on each request (ones that do follow through and load the full page).
> How many was it?
> Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time?
I'm intentionally leaving some of the details vague here, both because I don't want to be too harsh on the original author (it may have just been accidental/a buggy program/etc.), and because I don't want the traffic generators to know how im flagging them.
I wonder if this is why I see so many YouTube links posted under ‘new’ that never make it to the FP. Is this a way to get bot scraped and boost views/interactions?
Possibly, possibly not. I think some people are used to passing around news/information as short-form videos these days but HN (mostly) isn't into that kind of content.
Ok, on the one hand to have one bot hitting your website to populate some kind of slop newsletter is pretty annoying.
On the other hand, “multiple times per minute” doesn’t seem that high? Even if it was 60x per min that’s fully within the limits of what a 2015 laptop in the closet can handle.
Glad you're liking unslop.news :) If you ever find any bugs or want anything added, it's open source so feel free to open a ticket https://github.com/leiDnedyA/hn-without-ai
I don't know HN's rationale, but: if you try to ban self-promotion, then in practice you're only banning honest self-promotion. And if you allow it, but try to ban accounts that only exist for that purpose, then in practice you're encouraging them to spam mediocre comments in other threads to dilute the signal. [edit: I see that the guidelines do say not to use HN primarily for promotion. I'm willing to bet this significantly increases the amount of insincere engagement, but obviously the admins/moderators here are more aware of the tradeoffs than I am, and they have decided they are worth accepting.]
IMO the important thing to try to crack down on is vote manipulation; if you have that under control, then you can probably happily allow self-promotion and trust that only the interesting stuff will rise to the top.
Perhaps a contribution requirement per post? Though this proposal just invites bots to actually post comments because that becomes a requirement for posting links.
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
I assume that users who primarily/only post their own stuff and get flagged a lot reap the consequences, there's plenty of spam filtering on here thank goodness.
Those who slip promo of their own sites among a stream of other legitimate submissions or just space it out enough across multiple accounts can fly under the radar for years. People know how to game HN and already are, e.g. https://news.ycombinator.com/item?id=49517728
make interactions with others' posts/post others' stuff a soft req for posting your own stuff? So not following it could be deemed actionable by the hn mod team?
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
I _also_ posted to HN and made it to #1[0]; the effect this had on my CPU/network traffic on a 5/EUR month VPS over at Hetzner was in the single digit percentages.
As per the wiki page for the C10k problem[1] where it says "_was_ [emphasis mine] the problem of optimizing computer networking stacks to handle a large number of clients at the same time". This was solved over a decade ago...
The c10k problem is a different problem. It relates to simultaneous open TCP connections at a server. It would take a very very large number of crawler bots to force open 10k connections, I suspect.
There is so much spam, abuse, and aggressive scraping that I've had to block:
and a handful of other ASNs on almost all of my properties. It would be one thing if it was simple scraping, but it's often genuine agentic abuse (just spamming for vulnerabilities* or scraping millions of pages with very little valuable or dynamic content). These clouds must get the abuse happening under their own roof under control.I banned Alibaba's Cloud two years ago and it still hits one of my domains 1 million times per day, despite every route 4xx-ing during that time.
Even Apple has aggressively ramped up its scraping in the past few months, but thankfully they do respect robots.txt.
It's frustrating because some of my real users route through these clouds via their work VPNs. Also, there are a lot of less-than-reputable AI scrapers using global bot-nets via residential IPs to scrape, making it even harder to detect and ban them without affecting real people.
* like leaked .env routes, brute forcing admin panels, and so on
Bots and spiders hang onto old URLs longer than I hold onto a grudge.
We had some SSR that followed up with JavaScript listeners to do interactive updates instead of loading a whole new page. But the team responsible didn’t add nofollow and some bots ignore it anyway. Some significant fraction of our cluster size was answering those requests. But one deployment someone broke the fuck out of those links and it showed up immediately in our 4xx error statistics and we rolled the change back within a week.
Three years later I was still seeing those busted URLs being submitted.
Then the fix for the nofollow got implemented and declared done but the flag never got made permanent. We were pretty peeved when someone learned that in some offhand comment in a conversation between teams.
We were already identifying “good” and “bad” bots at the border and telling them to fuck off, so all of this was coming from our supposed friends. And that’s when I realized that a company owning a search engine and a cloud service provider is a protection racket. Google cloud is selling you servers in order to be talked to by Google bots.
> I banned Alibaba's Cloud two years ago and it still hits one of my domains 1 million times per day, despite every route 4xx-ing during that time.
You may as well silently drop the packets if they don't adapt.
Having the same issue with Bing bot. No idea why it's so eager to not behave.
Microsoft has never understood consent
Would you like your website to be scraped? [[Yes]] [Try again in 5 minutes]
Gates is in the Epstein files after all. Most organizations take their direction from the top.
> spamming for vulnerabilities
This has been going on for FAR longer than LLMs have existed. We call it "internet background radiation", because it's everywhere; it usually shows up within minutes of a host coming online. Or just about immediately after your first LetsEncrypt certificate gets issued.
Pretty much every infosec company of non-trivial size runs one set. The old venerable services are Shodan and Censys. Then there are a various security "researchers" who like to point their crapware vulnerability "scanner" at a subnet or ASN and let it rip.
And then there are of course all the actual botnets and low rent access broker wannabes trying to pop shells where ever they can, so they can resell that access.
Application-level banning seems very inefficient, these domains should be banned at the network (IP) level. Just drop any incoming packets from these domains/IPs, send absolutely nothing back that even confirms that you exist.
They are banned at the edge in Cloudflare (now), they never hit my application.
From a domain registry operators point of view Cloudflare are the worst of the bunch by orders of magnitude.
Google DNS are busy but its a hand full of IPs and reasonable. Cloudflare is non-stop never ending traffic. I don't know WTF they are trying to get. I've also seen TCP queries from those same Cloudflare IPs so this is not the result of a DDOS attack on them.
> Cloudflare is non-stop never ending traffic. I don't know WTF they are trying to get. I've also seen TCP queries from those same Cloudflare IPs
1. Breach router or desktop that hasn't been patched in ages via mass script
2. Establish C&C using cloudflare tunnels
3. Sell these hacked systems as residential proxy endpoints to everybody from state actors to teenagers
4. Some of them will be data scraping services
You are chasing your tail trying to ban by ASN or IP. It may work for another couple of months, at best.
Makes one want to build a parallel web based on web of trust.
This - but unironically. It has been striking how fast the public Web has descended into bot wars. I think something new will probably start to take over relatively soon.
People who used to contribute valuable content have pulled it because the bots were scraping it and reusing their work for free. Of course that was always technically possible but it used to be considered bad manners and not enough people did it to break the whole model. That's evidently no longer the case.
People who want to access real content are now interrogated with the assumption that they're probably a bot before they can even see a basic web page. Those interrogation systems are themselves are now running through a very small pool of providers with far too much influence and control to be healthy.
One of the most valuable means of communication and shared knowledge we have ever invented is being effectively destroyed before our eyes. I don't know what the solution will be - though I fear that the idea of privacy and anonymity online will suffer serious collateral damage - but I suspect that some new information network only available to people willing to identify themselves as a specific real human being in some relatively robust way will be a big part of it just as many sites are now gating their content behind user accounts.
I wonder whether the latter might be sufficient: have your terms state that user accounts are only available for real people and by using any automated system to attempt to create one the operator agrees to pay the site operator some ludicrous amount of money in recognition of the damage to their operating model that is posed by deceptively allowing automated access. And then make sure that such terms actually stand up in court - by legislating to enable them if necessary - in enough of the world to kill the AI scraper/bot model.
Cloudflare do a great job, I noticed moving from another VM.
And they offer free hosting!
at controlling internet sure
Can you do something with fail2ban? I.e. after a small number of requests resulting in 401 or 404, or a series of requests that's too fast to be reasonably from a human, ban that address for 48 hours.
If a legitimate request later comes from that address, it will probably get through.
But not sure how well iptables handles checking every connection against a list of possibly hundreds of thousands of addresses.
Fail2ban doesn’t work well against residential bot nets. A single ip generally only makes a couple of requests before rotating.
All those boomers out there with free IPTV just don't understand why it's free, my in-laws are guilty, but hey, they can glaze over and mindlessly melt like it's the 80s in different timezones and across continents
ipsets can handle vast numbers of addresses. I'm sure fail2ban uses them these days.
Does that old slowloris thing work but in reverse? Up your connection limits and keep N alibaba ones open forever, drip feeding them a small stream of data until they leak or timeout.
> These clouds must get the abuse happening under their own roof under control.
Similar to ad click fraud and scam ads, the cloud provider gets paid, why would they turn down the money?
So people accessing the internet through their cloud don't get automatically ip-banned by most services, perhaps?
Most big services will just take the junk traffic. They're never going to block whole cloud providers.
Blocking cloud providers is strictly for us dirty self hosters who are sick of our logs becoming unreadable and our visitor metrics useless from all this malicious traffic.
Does the money add up for that to be a deterrent? Are those using them for general VPN access paying enough that they'd care much, and if so is that enough compared to what they'd lose by kicking off the users abusing external sites (they can't all be using just free plans, and anyway tests to kick them could have enough false positives to lose them good paying customers as well as bad users). In any case is the amount of traffic enough for them to care to spend the effort tracking it to identify the bad users?
If advertising and tracking was less egregious, I wouldn't feel the need to use an ad blocker. But, here we are.
Sorry, what's the connection with the topic at hand?
[dead]
We spent a decade wasting cryptocurrency compute and still didn't come up with the decades old promise of a partial payment plan for web use. Seems like it'd make a lot of sense now that ad revenue is basically garbage and repetitive traffic is clearly just leeches and parasites that should pay for freeloading.
Someone hack a payment gateway into every webserver and MCP and let's get the bloodsuckers to pay for the internet.
> like leaked .env routes, brute forcing admin panels, and so on
That sort of thing was very common long before agent based abuse scanners. Volume would wax and wane but never went away.
Though other scans for vulnerabilities have shot up in frequency now everyone script kiddie and their flee-bitten mutt can ask an AI to do it for them.
This is a very smart thing to do! Thanks.
Why allow random requests coming from the clouds?
However, they could just use residential proxy botnets.
I believe the future is static resources with dynamic apps being only offered to people who can authenticate. And even then I’d recommend grandfathered accounts (eg telegram users that had an account for at least a year).
> However, they could just use residential proxy botnets.
Agreed
> I believe the future is static resources with dynamic apps being only offered to people who can authenticate.
I don't know about static resources, but I do think that identity will come into play.
I’ve had to block similar on clients sites. Their search pages would start to get hammered with obvious agentic search spam. Searching for thing not even related to the site in all combinations of the queries. Some of the user queries contained potentially PII.
Meta went absolutely haywire on one of my sites a couple months ago.
Is there a handy list of all of this that users contribute to?
so you're saying I should use my smtp filters on my website? (to be fair that seems like a good idea)
AWS is terrible. We had another tenant on AWS attacking us on AWS. AWS support took nearly a week of slowly bouncing around a case to do anything.
[flagged]
This is a very smart thing to do! Thanks.
Why allow random requests coming from the clouds?
However, they could just use residential proxy botnets.
Isn’t scanning for vulnerabilities illegal? Traditionally security researchers had to be very careful about such things
If there's a know CVE in a version of cPanel ir wordpress for example, those platforms broadcast what version they're on, so the matching cve being known is not illegal. You're just correlating known vulnerable versions without doing any probing. Pen testing however, a bit more involved in pulling info out without permission is illegal.
Can you enforce it? Would you even know who to go after?
Traditionally cybercrimes have been prosecuted by the victim going to their local PD (or the FBI) with the IP address of the offender, and then law enforcement subpoenas the ISP associated with that IP address to figure out the customer associated with that account, and then they have a case for prosecution. It all works as long as cybercriminals are less numerous than law enforcement, fall in the same jurisdiction (or in jurisdictions that have reciprocal extradition treaties), and have a real identity that can be tied to their online activities.
Now, many law enforcement agencies are tied up prosecuting political opponents of their bosses. To the extent that they have bandwidth to investigate and prosecute ordinary crimes, they're going to prioritize the ones that target a.) the government that backs them (of which there are many - you see another headline about the government being hacked every week or so on HN) b.) major corporations that have sophisticated IT systems that can aid prosecution or c.) headlines that will get their boss a PR boost and themselves a promotion. Your personal website does not register among those priorities. You probably can't even prove damages from the intrusion itself. The perpetrator might be a Russian bot that the U.S. doesn't have jurisdiction over, or a Canadian teen that the U.S. previously might've been able to prosecute with reciprocal judicial treaties but now has no leverage over because we've blown up our relationship with Canada. It may be legally a crime, but economically it's not worth prosecution.
Traditionally security researchers had to be very careful because they were doing everything above-board, with their real identity attached to their actions, and they made a very attractive (and PR-worthy) target. Somebody who's port-scanning from Russia through a VPN or residential proxy network is a.) anonymous b.) untraceable c.) out of U.S. jurisdiction and d.) not a headline-worthy target anyway.
A couple years ago I predicted here that a lot of things will break as the U.S. transitions from a high-trust to a low-trust society. This is one of them. Much of our justice system works well when criminals are rare and utterly falls apart when they're ubiquitous.
Last year, I wrote a post about how I use zipbombs to deal with bot traffic [0]. It went mega viral over the web, and I regret it. My server was completely overwhelmed with bot traffic. After months of dealing with it, I had to find a better strategy then keep my mouth shut.
[0]: https://news.ycombinator.com/item?id=43826798
Last year, I read your post and implemented it. I had no way to tell whether there is any benefit to it, so I discontinued it. Serving the data, even though it is compressed and the compression is precomputed, was still a waste of bandwidth. It could hit the browsers of legit user (false positive).
The main problem on my site is that bots hit the CGIT repo, where visiting files triggers syntax highlighting scripts.
I put in a skill-testing-question type system where you get a maximally long-lived access cookie if you answer right (so legit users are bothered as infrequently as possible).
Have you thought about adding a proof of work challenge? Your bandwidth is spared and attackers get increasingly expensive computational puzzles.
Disclosure: I work for Friendly Captcha, which uses the above strategy.
Your syntax highlighting is neither client-side nor pre-rendered?
CGIT doesn't support client-side syntax highlighting, as far as I know. Anything could be hacked, but out of the box, it has a system consisting of dispatching a script.
I have custom syntax highlighting. For instance, I have a home-grown script which takes a man page file (identified by a suffix like .1) and renders it readably, so it looks like a man page on your terminal, rather than nroff source code. (The only unsolved problem is that CGIT renders a column of the original line numbers down the left, which exceeds the length of the rendered page.) I also have accurate syntax highlighting for a home-grown programming language. That syntax highlighting exists only for Vim, which has a superb syntax highlighting engine. So in fact Vim is launched, via expect, to do some HTML rendering! There is no way that is getting translated into client-side.
Pre-rendering is not really an option over numerous git repos, that have numerous files in them, with the added dimension of history. What CGIT has instead is tunable caching.
Bots which aggressively scrape the history are hostile to the caching scheme, however. It works best for human visitors which mostly look at the head of the master branch in each repo, and poke around just a little bit in the history or look at other branches.
So it ended up not having a noticeable impact on the bots?
It worked fine... on a small scale But it's one thing to swat flies versus fighting swarm of yellow jackets. Serving a 1MB file a million times a day on a small digitalocean droplet is not feasible. Let's just say it didn't scale. And AI bots usually come with beefier machines, and recover faster after a bomb.
Why did you serve it from DigitalOcean?
Why not a CDN?
It's the content of the page, the raw html, that is replaced with a zipbomb. Example: https://example.com checks if you are a malicious bot, depending on the result, it will send you gzip html or gzipped bomb. So that can't be swapped or redirected to a CDN. The bots would simply ignore the redirect.
That sounds really annoying. Thanks for explaining! I'm looking to launch one of my new little products and my last one got hammered by bots.
Thankfully they were not really malicious like some of the other folks have experienced, but it was still frustrating.
I have a small website with a contact form that gets a lot of spam. I found a very effective way of filtering spam, but it only works because it's not mainstream, so I can't ever share it :(
Can that technique be applied to packet compression?
What is that improved strategy? :pray:
Not telling you this time, Mr. Bot
I really recommend blocking all traffic from anything on https://github.com/borestad/blocklist-abuseipdb
Thanks for posting this, as a paranoid self-hoster, I'll be adding it to my already extensive set of block lists.
I posted a link to an online game I made in the comments here once, and immediately got all kinds of good and bad traffic.
Some people liked my work, found me on LinkedIn and sent a connection with a positive message. Other people tried every which way to SQL inject or attack the site, posted slurs in every available input, etc.
Someone copied the site and hosted their own version with a couple minor changes. I only found that one because they had left one of the links in, which gave me referring traffic from their copy.
All that from one HN comment, very much pre-AI too, not to mention the standard daily slew of requests probing for what I assume are known vulnerabilities in Wordpress or PHP.
The internet is a floodgate of different actors, many of which are automated, and a link posted somewhere like HN opens the valve especially wide.
[flagged]
It's too late to comment on your older posts, but I'd just like to say: it's nice to see someone making oldschool web browsers work properly on the modern web.
https://www.sammystraus.com/#using-llms-to-make-the-emacs-we...
:) ive been dreaming of a usable emacs browser for so long
I was going to comment on that too. Talk about burying the lede! The idea of using an LLM to generate a static program for a specific web site is brilliant. It reminds me of the second Futamura projection.
https://en.wikipedia.org/wiki/Partial_evaluation#Futamura_pr...
https://blog.sigfpe.com/2009/05/three-projections-of-doctor-...
I rolled my own version of Anubis for reasons*, and > 80% of all traffic is bots that fail the challenge, even when white listing legit crawlers.
* I wanted an elixir version so I could run it alongside my existing apps instead of having it sit as a proxy in front of them. This way my existing apps can validate if the caller is already whitelisted and if not redirect to the challenger https://sphinx.8by3.com which sends you back to a site if you pass eg. https://tuhat.net
If you are rolling your own it probably doesn't need to be as clever as Anubis is trying to be. Simple something relatively CPU heavy in JS will do the trick, bonus points if you can make it memory-heavy too. Most bots will give up running something CPU heavy currently (though they will run a lot of JS stuff as much of the web is inaccessible without) and they aren't going to optimise your challenge in WASM or similar because it isn't worth it for the small number of sites it protects (Anubis' challenges on the other hand may well be worth optimising for that way because of their wider use, hence that project is stuck in the cleverness war of attrition and definitely needs to move toward memory-hard problems and something more efficient than JS sooner rather than later).
My main reason for planning to implement something like that myself instead of something off the shelf is two-fold:
1. The points mentioned above
2. I can be a dick and make some subsets do harder work (longer challenges, adding time delays that wouldn't phase a bot) - “hi there North American IP, I know how much you like extra hassle hence the love of effectively paying for tarriffs, here, have a little extra work on me!”
The challenge isn't terribly clever, just some basic crypto maths, same as Anubis. There are a very low number of AI bots that pass it based on my logs, like 0.01% low and I suspect they are interactive agents that bother to run JS as crawlers typically don't. Assuming these are actually run on an end users computers I don't think any amount of cpu/memory weight would be a real blocker. One thing I've seen that is interesting'ish is bots failing as they answer the challenge from a different IP than it was requested from, and the source IP is validated as part of the response so they fail.
> that bother to run JS as crawlers typically don't
I'd be surprised if many don't run basic JS, given how many sites require it for basic content these days (load base site, then make a fetch request for the actual content), unless there are less such sites than I think.
> One thing I've seen that is interesting'ish is bots failing as they answer the challenge from a different IP than it was requested from, and the source IP is validated as part of the response so they fail.
From what little I've looked into the details, it seems that scrapers (both old and new) using a scattering arrangement (built in or via an outgoing proxy) seem to be split 75/25 between those that completely scatter (following requests will almost certainly come from a different address) and those that stay on the same address for at least the first few requests (which would cover login, bot challenge, or such) then move around to avoid requests-per-source-address based resource management measures. Only really simple ones (someone running a recursive wget without proxy) stick to the same source address all the time now.
Unfortunately you can't be more strict about changing source addresses because of how many false positives you will see. CGNAT arrangements often move people between addresses fairly regularly, and even without that there is the common case of people moving away from a WiFi AP and automatically flicking to their mobile network (or another nearby AP).
> “hi there North American IP, I know how much you like extra hassle hence the love of effectively paying for tarriffs, here, have a little extra work on me!”
I'm assuming you're outside the US based on having this idea.
You might as well just say "I'm mad at your country for the tariffs, so here's an extra wait." From the fact you feel the urge to spite people in a tariffing country like this, it's clear the tariffs have had some negative effect on you, which goes against your narrative that they're "effectively paid for" by those in the tariffing country who voted for it.
You have a some incorrect assumptions there. Yes, I'm outside the US, though I can't say I've experienced any issues at all that can be attributed to the continuing little tariff spat. I have heard of US citizens either complaining about cost raising because of them, or if not complaining then acknowledging them as something they consider a necessary pain. There are a fair few things I'm mad about, some of which do stem from the current administration over there, but this certainly isn't one of them. In fact I'm currently amused by it and the urge to mildly irritate those state side is in order to extend that amusement a touch.
I see, you're supposedly just amused by annoying people... that certainly makes you look better, I can see now why I was downvoted. /s
If you think your nagware is going to get back at US citizens for completely unrelated policies you're mad about, then maybe you should list those policies in your message instead of bringing up tariffs? That seems like it would be more productive to your goals, and less likely to make it look like you're salty and/or making a faulty argument regarding tariffs.
> I see, you're supposedly just amused by annoying people...
Some people, yes. I'm not claiming to be a good person, especially not that this makes me one, in fact stated “I can be a dick and…”. Some of my interactions on social media are similar - I do it to vent a little, for a little dopamine hit & mental health boost, or just because I need a short escape from work/chores/other for a few minutes and the get-some-fresh-air, or play-with-the-cats options are not available at the time. I call it my “breakfasts with the Percy's”.
For the record I, except for very rare occasions, don't downvote: I either comment or move on. Most likely the people that have downvoted have done so not because they disagree with you (maybe they do, maybe they don't) but because that post was the point we stepped too far off-topic, so perhaps try not to take that personally.
> more productive to your goals
Why do I need to be working to a world changing goal beyond a bit of “isn't the regime over there a bunch of great steaming wazzocks?” banter that might amuse some people I like, and maybe irritate some who I disagree with¹? Some things are just little in-jokes, not everything has to be for grand effect in some more meaningful agenda.
--------
[1] though they'll likely never see it, this is stuff I only put online for me and a few family/friends, including a couple state-side. Even the scraper bots haven't found it yet (the look-for-known-vulnerable-software scanners pass by regularly, so that is likely only a matter of time) so I doubt a mass of random human visitors are ever going to flow in.
I want mini games! "Score 5 points" style. Although, it would surely fail ADA compliance. I don't see how any of this could be practically effective, respecting ADA.
In server logs you can distinguish the type of LLM's traffic. OAI-SearchBot/GPTBot/ClaudeBot is just indexing but ChatGPT-User/Claude-User etc. may be valuable traffic generated by user's conversation with LLM's. So it's worth choosing which ones you block, instead of blocking all of them.
> valuable traffic
What's the value for the website maker in these? There's user value to it (rather than just value for Anthropic/OpenAI) but the whole point of these bots is to make sure nobody ever needs to visit your website.
Tons of people click from ChatGPT! It's now the top referrer on most of my websites, and seems to be on path to overcoming Google soon (a ton of traffic from Google has vanished since the AI Overview rollout...).
Yeah, no wonder.
I realized its much faster to do the "google thing" just with chatgpt. Its filters all the sh!t for you. Searching the net frustrates me most of the times this days. Search engines are software for an older generation of the www.
In my case this is the main source of registered users. If a user is engaged in a discussion with chatbot and gets the recommendation from it his engagement is higher. For example, in my system, a user has registered and within one hour used the system 7 times and stayed active while for registrations from Google I can observe zero-activity accounts.
Strong agree, a good way forward is letting user agent once again truly mean an agent on behalf of the user. And speaking anecdotally, I've seen different traffic "shapes" under the same UA, which leads me to suspect that some actors might be disguising traffic as more mainstream labs.
Even if you block the user-agent, the traffic returns under a fake browser user-agent. No way to keep the bad ones out.
[flagged]
[flagged]
I’ve found that Claude is really helpful for finding patterns in my web logs and identifying bad traffic patterns. Meta was ignoring robots.txt on a wiki that I run and generating so much traffic that it was effectively DOSing the site until I blocked their crawler.
What’s been really handy is being able to identify “stealth” bot traffic by the request patterns (e.g., not requesting CSS/JS/favicon when loading the page).
[dead]
I got $10K GCP bill one month because of ClaudeBot. Its likely some users were trying to replicate our software by using Claude code + chrome plugin. If I was spending $10K on traffic, I can only imagine the money cloners must be spending on tokens.
Well, potentially very little! It could be subscription usage, or just really inefficient code authored by an agent. It probably costs LESS in tokens to write a poor quality cloner that breaks upstream things.
To solve this bot traffic issue first I check website Real traffic by using SerpSpur Bot Traffic Detector (TAI™ Checker). I can use this tool past 3 months and my client website is ok from fake traffic. Thanks!
Basic auth worked for me. Locked down everything but the home page. Put the credentials on the home page. No more shipping my assets to nowhere hundreds of times a day. Of course, I had _zero_ traffic other than bots...nowadays I have given up putting my crap out there for no one to see.
> I won't name names, but I looked them up, and they describe themselves as a service that listens for changes to a website, and then summarizes them into an AI slop news letter. And by 'listens to changes' I mean query our website many many times per-minute. Blocked.
That sort of behavior would justify a name and shame imo
Inconsiderate shitty services don’t deserve the politeness of protecting their name
I wonder if you could just blast them with ever changing content to burn their summarization tokens on BS.
Though they probably use the cheapest possible models they can so the damage from their dumb business model would be limited
I've been posting about this recently but I continue to be bad at either timing my submissions or titling them. I struggle with YouTube thumbnails too so I think it might be me. But anyway:
I experienced the same thing after submitting articles from my small photography website. I did debate using some of the off-the-shelf solutions like Cloudflare, Anubis, or iocaine, but I ended up building my own L7 proxy that utterly fucks the bots up: https://robertmay.photography/journal/bwbach-my-guardian-gob.... I will make it generally available at some point once I get it optimised how I'd like, but currently I'm able to trap about 3000 concurrent crawlers in my tarpit of slow requests on two small Hetzner VMs. And that 3k is really just the limits of the networks hitting my servers right now, I think it'll go significantly higher than that.
I haven't really seen anyone else combat them this way, possibly because many of the standard proxies and servers are set up for fast requests, not serving them as slowly as possible. Deliberately serving page loads for as long as the requester will accept packets turns out to be a really funny way to stop the botnets - when I just rejected the requests they in turn hammered my server even harder. By tanking the request volume with a ridiculous BEAM-based proxy you can reduce their request speed to a crawl and I'm fairly certain a couple of them have blacklisted my server now.
I like the spirit of this approach; make them suffer! But I'm surprised the bots wouldn't have reasonable timeouts to counteract. How long do they typically wait on these requests?
EDIT: sorry, that was a dumb question before reading your post. You actually learn the timeout they tolerate. Nice!
I should probably put together some actual stats at some point, but from what I see in the live logs you'd be shocked - some of them will happily sit there for over 5 minutes being drip-fed packets!
Looking at you, Meta.
I do still get some requests going up to 15 minutes which is quite ridiculous, though those don't seem to be super common (maybe they're learning).
I think the shortest timeouts I've seen are between 2-5 seconds, but they're surprisingly infrequent and actually that's still a massive inconvenience for them. I did at one point just drop any request I couldn't keep open for more than 5 seconds but it actually caused more server load. I'm essentially inverting the server load by making slow requests more efficient, which is counter to everything I ever learnt about performance for web serving :D
I think the average I see is probably somewhere between 15-30 seconds. I'm guessing many crawlers don't want to go too low because they'll be trying to fetch large files like images, or perhaps they know they'll be slowing servers down and want to ensure they get the response. Most likely, the majority of crawler authors aren't expecting some arsehole to exploit this and they've just left the setting as the default.
I'm running some tests currently
https://deep.43z.one (how deep bots will crawl)
https://stalling.43z.one (how long they'll keep the connection open)
Ooh that’s fun! I’ll have to add some stats like this to mine too. Anthropic keeping connections alive for 2 hours is wild. I don’t think they fall into my trap as I believe they respect robots.txt, which is at least an improvement on Meta.
> 3000 concurrent crawlers
I gave up on tarpits when it got to 130k "waiting" connections.
> one specific user. I won't name names
Who is?!
haha im not naming names
You shouldnt defend bad actors and abusers. Hiding them is tantamount to them attacking others indiscriminately. Cause youre not the first victim, and definitely not the last.
You see something, you say something, or you do something.
How else will the author play the victim for some clicks then?
Is it SpokenFeeds?
Does the country end with an "a" with 5 - 6 characters long?
Two things I do to slow down malicious bot traffic:
1. Setup a blackhole for bad bots. I don't use WordPress, but I stole the idea from this plugin: https://wordpress.org/plugins/blackhole-bad-bots/
2. Setup a Geo-IP allow list using MaxMind's country DB (https://github.com/P3TERX/GeoLite.mmdb). I only allow visitors from countries I do business with.
I'm trying to understand the reasoning behind the entity that was sending presumably hundreds or thousands of requests per minute to the same page. It's some kind of AI news site, so it makes sense they want the freshest content, but it sounds like they were doing many requests per second, so that can't be the reason?
> I'm trying to understand the reasoning behind the entity that was sending presumably hundreds or thousands of requests per minute to the same page.
The most likely answer is they don't give a crap because it's not their money and no one is going to do anything about it.
Nice. How do I get a massive swarm of nodes on someone else's money?
It's called a residential proxy botnet, and you can rent one if you have no morals,
https://www.lumen.com/blog/en-us/symbiotic-parasites-the-mod...
For free?
If it's an AI news site it's probably coded by AI, and by someone who doesn't know what they're doing, or what "rate limiting" is.
> my website is built as a single-page-application
There you go. Ditch it. Make you pages cachable as much as you can.
doesn't fix the issue
It does. Split read/write, host your read-only content on CDN, all worries gone.
That deals 95% of the headache
Some bots would click around and submit data, you need to deal with them anyway.
Don't waste your time on something easily scalable and cache-able. Let CDN/WAF providers do the job for you.
GitHub Pages et al offers best of both worlds.
Have you tried dealing with bots using CF Turnstile? Ref: https://www.cloudflare.com/products/turnstile/
I run a few private MMO servers, we have a website with a support forum, in order to post, you need to register, log in, fill in account data, then create a thread, each of these pages is protected by Turnstile as well as a general bot check for the website, we frequently get automated spam from RMT (real-money trade) services for the OG game. Turnstile is maybe effective against simple bots that just try to call curl/requests/whatever but if you have bots that use a browser driver it's completely useless.
[flagged]
Turnstile is no longer effective. Anyone with a budget uses a real headless chrome instance and gets around it. You only need the spendy chrome instances to get the initial cookie, then you pass that to your workers and they scrape.
It’s really starting to suck. The worst part is these are coming from consumer vpn providers that we can’t just block (lots of real users paying real money on our service are vpn users) or they come from big residential isps. The whole residential relay thing is a hard adversary, since it’s spread across so many real users paying ips that you can’t whackamole it. I’d say 1% of residential ips are participating in a botnet (there’s prob some real numbers out there if I cared to look).
This is just going to push everyone harder into walled gardens. Web input biggest inbound but our content is our mote, we are going to have to turn off non logged in content viewing, which will have real business impacts.
I did this for alyph.ai, but was recently flooded with so many browser-using, Turnstile-defeating bots from Taiwan that I had to completely turn off sign ups. It's not fun anymore.
Not even third party login worked (Google, Linkedin, ...) to curb this.
They were logging in, scraping the client API for vulnerabilities/possibilities and then spammed erotic dog poems to Claude Fable until the $5 welcome credit got drained.
Fun New World.
Since there were a couple upvotes, here's one I saved and Google-translated:
... [some more gibberish] ... Interestingly, in the markdown, which was Chinese mostly, "Phase" and <story-record> were in English.remove free signups
$5 minimum to sign up
not yet, but I think something like this would be a next step
[flagged]
It's unfortunate that this also just leads to port scanning and huge accidental bills. This happened in the past to more than one acquaintnace of mine; go and market their vibe-coded/OS project on HN + twitter and then suddenly the vercel bill goes through the roof with bots scraping the website to bits, even with bot protection from CF. Really hard to get away from it these days, but hopefully over the next few years some new startup will raise $50M for "killing cloudflare" and will actually follow through!
[flagged]
Oh, as soon as you add tls to your subdomain, the bots come. It’s instant.
Certificate transparency logs get scraped. However, they cannot scrape wildcards. Get a domain.example + *.domain.example cert and host your blog on blog.domain.example and you can keep the bots from finding you, at least for a little while.
No, it wouldn't. Just yesterday I added a domain.example + *.domain.example certificate, and today some bot comes to scan for subdomains: api, app, backup, beta, cdn, ci, mail etc. Blog isn't in the list, but just for this bot.
Come visit my website at F31A8A03-6FC9-40B9-BA4E-EDFA68E0E90A.clownpenis.fart!
What confused me most about this is that the entire site is, apparently, a single page (there's no sitemap that I can use to verify this; there IS an RSS file which appears to confirm it) containing all posts. Links at the top (headed "Table of Contents") are all in-page links.
At first, I assumed this was a single blog post, especially with a "Table of Contents" at the top. I guess it's following some old-school blogging conventions, but AFAICR, even those sites retained individual pages for separate posts.
it's org-html-export-to-html babyyy lmao
I think I could improve the format a bit with something like Jekyll, but I haven't had a good chunk of time to sit down and work on it
You can also link between Org documents and org-publish will export the documents with working web links, if you're lucky. That's how https://entropicthoughts.com works.
I've wanted to work with org mode for so long, but I can't simply bring myself to use emacs. Nothing personal. Org mode is awesome, tho.
if you do decide to give it a try, and are coming from vim, i've had a good experience with https://github.com/emacsorphanage/god-mode as a way to kind of bridge the gap into modal editing
I am trying to do the same thing, but on purpose for https://notforhumans.pages.dev/
I have only had to block Lightpanda user-agent for super aggressive scraping from a multitude of source IPs. I did have to implement heavier rate limiting for some application paths to reduce performance impact from other agentic traffic. Rather than wholesale blocking agents I’d prefer they see my applications as useful resources and drive potential premium api upsells to agentic developers who want bigger rate limits
I used to roll my own (respectful) scrapers. AI and vibecoders have ruined it for me. Everyone has super sensitive rate limiting and not checking now.
I posted a link to HN a few days ago that got literally zero upvotes and zero comments (Applying NASA's satellite-imaging trick to ancient museum objects https://news.ycombinator.com/item?id=50023128) and it was hit with insane bot traffic. HN was the only place I had posted it.
The output side is just as lopsided. Google alone was at ~3.2 quadrillion tokens/month in May: roughly 79T words a day at 0.75 words/token, against ~100T words/day humans produce.
It happened to me too. I got a big cloud egress bill because my game[1] is basically a mirror of thousands of Wikipedia articles. AI bots would smash all the pages so my cache hit rate was low. [1] https://redactle.net (come at me Claude)
I get llm-like traffic on domains with zero web content (never was, probably never will be any).
Just merely existing will get you scanned these days.
If you have an SSL certificate so such a domain can serve HTTPS traffic if needed, which it wouldn't surprise me if some parking services generate for you these days via an ACME provider, then your domain will be in the certificate transparency logs so can be found that way.
One possible way around this is to use a wildcard and not serve anything on the base domain. Unless the right subdomain is listed somewhere, they'll not get any scans. Maybe useful for projects that you have online for just yourself and a small closed circle.
if you want the site to be fast, don't render it with javascript and take a look at the google lighthouse check for things to improve:
https://pagespeed.web.dev/analysis/https-sprinklz-io/bmc4mv1...
That's a vague and borderline off-topic suggestion, imo. When the problem is too many malicious visitors hammering your server, the amount of time it takes the visitors' browser to run the JavaScript is not the issue; on the contrary, when dealing with this issue, you might as well do more rendering client-side to offload as much work from the server side as possible (hence the Anubis-esque defense of adding random JavaScript work that isn't even necessary to render the real page).
I have tried a ton of stuff (Cloudflare, Anubis, fail2ban...) to combat the bots lately, but they all have their downsides, and users complained too much, so I just gave up. You simply can't win this arms race.
So I have resorted to performance improvements on my server and serving every single request. Things are way smoother now.
Try posting something viral on Mastodon. The logs clearly show that it isn't the most efficient social network.
Can you elaborate more, please? I'm curious.
I recently posted a link that got 2 boosts and 6 faves, but it was requested by about 300 fedi servers so they could generate a preview.
Which is fine! Because even though my photo gallery runs on a 15 year old laptop in the basement, I wrote the gallery software in a fast language (Rust) and so a trickle of requests maxing at 4 per second is not actually a problem.
People complaining about fedi hammering their servers are probably using PHP or Python.
Would even Perl 5 CGI pages have problems serving that load on a modern CPU? Doubt it.
Probably not! I don't know what the deal is.
not the parent commentor, but I've previously heard complaints about how each server separately does a fetch of images and html for previews.
Meanwhile if you publish on GitHub pages and aren't notable you will apparently get barely any traffic at all :( (Unless the bots are somehow just deciding not to follow my third-party hit counter links, anyway…)
Wow, your product is a smart idea! But never really thought that these things were caused bya service turning websites into AI newsletters. It's interesting to me that this can cause the website to be slowed down.
> somebody is trying to access -/wp-content/phpinfo.php
This is normal, I see hundreds if not thousands of these every day in my logs. You can use fail2ban to filter out some of the noise, but ultimately what you can do is limited.
Oh is that why I keep getting those ai emails about the project I posted earlier?
Yeah it's awful they keep coming months after you post anything as a Show HN now, definitely discourages me from doing it
One option to reduce it, though this is work to setup initially:
1. Use a catch-all subdomain
2. Reject email to addresses where a simple checksum does not match (valid addresses being something like asdfghij.klmn@catchall.domain.tld where the klmn is taken from hash of asdfghij and some secret, anything not matching that pattern gets rejected) [this means running your own mail server so you can do that on incoming mail, running a scan to pick out and remove errant messages regularly, or having a mail client where you can implement this as a plugin]
3. On your contact detail page, don't give out a straight email address but have the the page run some Anubis style JS before asking the server to respond with an address that works
4. Once an address is obviously being used in a manner you don't care to respond to, add it to an explicit block list.
For extra protection from addresses being passed around (intentionally or via hack & other leaks), ask for their email address first and use that as part of the hash, then you can reject mail to your catch-all that doesn't come from the original other party.
Whether all this is worth the effort of implementing, is of course up to you. It probably isn't… I've been considering something like it, but mainly as a fun little project rather than something that would be massively useful.
Wait until you answer in "Ask HN: Who wants to be hired?" and start receiving hiring phishing emails.
This happened to me. My site got shutdown for awhile because of the traffic after I posted an article on HN. I think they are trying to DDOS me (puts on Tin Foil Hat)
Interesting, thanks for sharing.
>So I took a look at some data. I added top-level logging for every endpoint (how many times was a request made to each url path), and I found that a majority of the requests did not even make an attempt to fetch news articles! For reference, my website is built as a single-page-application, and what this means is that a majority of the requests did not even run the javascript necessary to render the page (!).
So you got hammered by curl requests! But isn't that part of your delivery static anyway? Were you regenerating the same HTML over and over?
>many, many requests per minute?
How many was it?
Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time?
But hey. Silver lining: free load testing! ;)
> So you got hammered by curl requests! But isn't that part of your delivery static anyway? Were you regenerating the same HTML over and over?
i dont think this was the cause of the slowdown, it was just sort of the thing that led to the discovery that this was mostly bots.
in general, because the purpose of the website is to allow users to generate news feeds based on custom criteria, it does do a bit of meaningful work on each request (ones that do follow through and load the full page).
> How many was it? > Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time?
I'm intentionally leaving some of the details vague here, both because I don't want to be too harsh on the original author (it may have just been accidental/a buggy program/etc.), and because I don't want the traffic generators to know how im flagging them.
Threads like these are what always reset my momentum towards “real” self-hosting of anything public-facing. It sounds like a nightmare these days.
I wonder if this is why I see so many YouTube links posted under ‘new’ that never make it to the FP. Is this a way to get bot scraped and boost views/interactions?
Possibly, possibly not. I think some people are used to passing around news/information as short-form videos these days but HN (mostly) isn't into that kind of content.
I assembled some crons to pull my access logs, process through LLM and notify WHOIS abuse contact.
That is primarily why websites only TLS-negotiates for server-side-only ChaCha-Poly.
How could the specifics of TLS help deter/block bots?
Different Implementation of HTTP requests modules.
I ran into similar problems. My analytics became so useless that I threw them out.
The same thing happens every time we post here, content is ignored and bots swarm. We started blocking whole IP ranges.
Created a long rant with some solutions and free tool Vorwatch: https://vektormemory.com/opensource
Unpaywalled article: https://medium.com/@vektormemory/bots-lots-of-them-5a1e35be2...
Same with me, every-time I got 1st page I got a lot of robots
i banned everything from outside of germany. But i also get robots from everywhere
love the simple css of this. very tasteful and beautiful, even if its just simple.
Heh. I posted a GitHub project Show HN, a couple of days ago, and it never even made it out of new, but it got me some bots.
That's a big reason that I never post any of my apps here.
Welcome to hell, kid...
Ok, on the one hand to have one bot hitting your website to populate some kind of slop newsletter is pretty annoying.
On the other hand, “multiple times per minute” doesn’t seem that high? Even if it was 60x per min that’s fully within the limits of what a 2015 laptop in the closet can handle.
yeah, it was a wake-up call that I needed to make certain parts of my application faster, and I've been working on it for the past few weeks
Once every 24 hours seems sufficient for the stated purpose.
groped to death
happened to me too!
I’ve been enjoying https://www.unslop.news/ after it was shared here.
HN has become unbearably AI slop oriented over the last couple of years. It’s nice to see something else.
Glad you're liking unslop.news :) If you ever find any bugs or want anything added, it's open source so feel free to open a ticket https://github.com/leiDnedyA/hn-without-ai
same, but I see degraded quality in the submissions thatbare left, do you experience the same?
[dead]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
[dead]
[dead]
[flagged]
[flagged]
I don't know HN's rationale, but: if you try to ban self-promotion, then in practice you're only banning honest self-promotion. And if you allow it, but try to ban accounts that only exist for that purpose, then in practice you're encouraging them to spam mediocre comments in other threads to dilute the signal. [edit: I see that the guidelines do say not to use HN primarily for promotion. I'm willing to bet this significantly increases the amount of insincere engagement, but obviously the admins/moderators here are more aware of the tradeoffs than I am, and they have decided they are worth accepting.]
IMO the important thing to try to crack down on is vote manipulation; if you have that under control, then you can probably happily allow self-promotion and trust that only the interesting stuff will rise to the top.
Any thoughts on what the alternative would be?
Perhaps a contribution requirement per post? Though this proposal just invites bots to actually post comments because that becomes a requirement for posting links.
The guidelines already cover this:
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
https://news.ycombinator.com/newsguidelines.html
I assume that users who primarily/only post their own stuff and get flagged a lot reap the consequences, there's plenty of spam filtering on here thank goodness.
Those who slip promo of their own sites among a stream of other legitimate submissions or just space it out enough across multiple accounts can fly under the radar for years. People know how to game HN and already are, e.g. https://news.ycombinator.com/item?id=49517728
That one didn't work though:
https://news.ycombinator.com/threads?id=FannyJoeStyles
That's just one account that unmasked itself and was noticed. Doubtlessly that didn't stop them from moving on to the next one.
Oh, absolutely. But there are accounts that are active here for months or years, that one at least got killed while still green.
make interactions with others' posts/post others' stuff a soft req for posting your own stuff? So not following it could be deemed actionable by the hn mod team?
Edit: nvm thats already in the guidelines
Ostensibly, they aren’t. From the Guidelines (https://news.ycombinator.com/newsguidelines.html):
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
I _also_ posted to HN and made it to #1[0]; the effect this had on my CPU/network traffic on a 5/EUR month VPS over at Hetzner was in the single digit percentages.
As per the wiki page for the C10k problem[1] where it says "_was_ [emphasis mine] the problem of optimizing computer networking stacks to handle a large number of clients at the same time". This was solved over a decade ago...
Now get off my lawn!
(TBF I think the author acknowledges this)
[0] https://news.ycombinator.com/item?id=43725815
[1] https://en.wikipedia.org/wiki/C10k_problem
The c10k problem is a different problem. It relates to simultaneous open TCP connections at a server. It would take a very very large number of crawler bots to force open 10k connections, I suspect.
> It would take a very very large number of crawler bots to force open 10k connections, I suspect.
You can do that with any system made in the past 15, maybe 20 years.
From the wikipedia page:
"As of 2025, the problem has long since been solved, with the number of possible connections to a single computer being in the millions."
I know it's a different problem but presumably there's at least some relationship/order of magnitude being expressed here...oh well