Naive question: services that can be used for amplification attacks, are they constantly getting patched to prevent the latest iteration of attack type?
In other words, if there are a bunch of services prone to amplification attacks, can traffic from these services be upstream-blackholed for the duration of the attack?
If it's not traffic coming directly from an IoT botnet, which is probably where the source of the spoofed traffic that initiates the amplification, then isn't there likely a smaller, more manageable number of services responsible for the attack traffic?
Or are we talking services that form the substrate of the internet that have inherently exploitable protocols that it would take a large herd of organized cats in order to update in a way that doesn't break the internet, and will still take ~10 years?
I still think in IPv4, so this may be a stupid question, but it's it known how many unique IP addresses were attempting to connect in the space of that time, and then it's there logging to identify those with unusually large amounts of individual traffic?
Ten years ago, when I worked on stuff that attracted DDoS, the memorable vectors were UDP chargen reflection and wordpress pingback reflection.
Wordpress does get lots of patches, but I don't know how you really fix pingback, but it was easy enough to look for user-agent WordPress and drop requests before serving large files (or really anything... what do I have that WordPress should request). For a smaller site, the volume might have been high enough to overwhelm TLS handshaking, which is harder to solve.
Chargen, wow. There's pretty much zero need for it to be on the internet. There's no need for anyone to run it. But evidence showed many instances running and it seemed to be the implementation Microsoft shipped in the Services for Unix package. Someone was trying to blocking the reflected traffic, but the servers were sending 64k responses (!) and that was being fragmented, and they only dropped the first fragment... fun times.
I didn't spend time trying to get the hosts involved to stop sending this garbage... Writing abuse reports is herding cats, and networks that would be responsive probably already have taken these senders offline. I was also seeing short duration attacks consistent with people trying the free tier of DDoS as a service... so dealing with 90 seconds of garbage every once in a while was no big deal (as long as fragment reassembly didn't knock the machine over)
> There was no unauthorised access and no compromised systems. This was an overload attack, not an intrusion.
Hopefully your logging infra is rock solid and nothing has been dropped in the flood. It wouldn't be the first time a DOS was used to mask the actual attack by overwhelming the monitoring infra.
> Use a CNAME or ALIAS record instead of an A record. An A record ties your domain to one specific IP address on our platform. That fixed binding was exactly the problem during the attack: wherever we could change the address on short notice, availability could be restored, wherever we could not, only the blunt measure remained.
I don't understand how this helps. CNAMES have TTLs like A records and they eventually have to terminate at an A record somewhere so why pay for an extra hop?
> I don't understand how this helps. CNAMES have TTLs like A records and they eventually have to terminate at an A record somewhere so why pay for an extra hop?
I assume the customer controls the domain DNS records here rather than the hosting provider.
CNAME record: hosting provider can change underlying IP freely.
A record: hosting provider must get in touch with customer to change DNS.
I sometimes work on a system so byzantine and poorly-documented with so many horribly obsolete systems - they have Windows XP machines all over the place - that there's no way it hasn't been popped.
I think it's safe though.
I think various groups have popped it, taken a look around with growing expressions of horror on their faces, and gotten back out as quickly as they could. Nah man, no way, not touching that, don't want my prints on this particular gun, there's no way this is legitimately how they've left it, this has to be some kind of a setup.
Yes, that's bunny.net, it's named in the post. Their Zurich PoP appears to sit on CDN77's network, which is on their side of the setup, not something we configured.
Yes. The blog version was drafted with AI assistance and then edited and reviewed by us. The facts, numbers and timeline are ours and match the technical postmortem PDF linked in the post, which is the drier, more complete version. Happy to go into any of the technical details here.
Naive question: services that can be used for amplification attacks, are they constantly getting patched to prevent the latest iteration of attack type?
In other words, if there are a bunch of services prone to amplification attacks, can traffic from these services be upstream-blackholed for the duration of the attack?
If it's not traffic coming directly from an IoT botnet, which is probably where the source of the spoofed traffic that initiates the amplification, then isn't there likely a smaller, more manageable number of services responsible for the attack traffic?
Or are we talking services that form the substrate of the internet that have inherently exploitable protocols that it would take a large herd of organized cats in order to update in a way that doesn't break the internet, and will still take ~10 years?
I still think in IPv4, so this may be a stupid question, but it's it known how many unique IP addresses were attempting to connect in the space of that time, and then it's there logging to identify those with unusually large amounts of individual traffic?
Ten years ago, when I worked on stuff that attracted DDoS, the memorable vectors were UDP chargen reflection and wordpress pingback reflection.
Wordpress does get lots of patches, but I don't know how you really fix pingback, but it was easy enough to look for user-agent WordPress and drop requests before serving large files (or really anything... what do I have that WordPress should request). For a smaller site, the volume might have been high enough to overwhelm TLS handshaking, which is harder to solve.
Chargen, wow. There's pretty much zero need for it to be on the internet. There's no need for anyone to run it. But evidence showed many instances running and it seemed to be the implementation Microsoft shipped in the Services for Unix package. Someone was trying to blocking the reflected traffic, but the servers were sending 64k responses (!) and that was being fragmented, and they only dropped the first fragment... fun times.
I didn't spend time trying to get the hosts involved to stop sending this garbage... Writing abuse reports is herding cats, and networks that would be responsive probably already have taken these senders offline. I was also seeing short duration attacks consistent with people trying the free tier of DDoS as a service... so dealing with 90 seconds of garbage every once in a while was no big deal (as long as fragment reassembly didn't knock the machine over)
> There was no unauthorised access and no compromised systems. This was an overload attack, not an intrusion.
Hopefully your logging infra is rock solid and nothing has been dropped in the flood. It wouldn't be the first time a DOS was used to mask the actual attack by overwhelming the monitoring infra.
> Use a CNAME or ALIAS record instead of an A record. An A record ties your domain to one specific IP address on our platform. That fixed binding was exactly the problem during the attack: wherever we could change the address on short notice, availability could be restored, wherever we could not, only the blunt measure remained.
I don't understand how this helps. CNAMES have TTLs like A records and they eventually have to terminate at an A record somewhere so why pay for an extra hop?
> I don't understand how this helps. CNAMES have TTLs like A records and they eventually have to terminate at an A record somewhere so why pay for an extra hop?
I assume the customer controls the domain DNS records here rather than the hosting provider.
CNAME record: hosting provider can change underlying IP freely.
A record: hosting provider must get in touch with customer to change DNS.
I probably incorrectly read
> we provide that customer’s connection to the internet
as providing the DNS infra for their customers.
"We found three concrete gaps during this incident, and we would rather be upfront about them than gloss over them." claudism?
Sticks out like a sore thumb, doesn't it?
> There was no unauthorised access and no compromised systems. This was an overload attack, not an intrusion.
"AI said it's all good. There are no attackers within our walls."
I sometimes work on a system so byzantine and poorly-documented with so many horribly obsolete systems - they have Windows XP machines all over the place - that there's no way it hasn't been popped.
I think it's safe though.
I think various groups have popped it, taken a look around with growing expressions of horror on their faces, and gotten back out as quickly as they could. Nah man, no way, not touching that, don't want my prints on this particular gun, there's no way this is legitimately how they've left it, this has to be some kind of a setup.
Security by obsolescence layered with security by complexity.
Enterprise security in depth.
it's not enough to salt your hash; you must mold your Swiss Cheese
Yes, that's bunny.net, it's named in the post. Their Zurich PoP appears to sit on CDN77's network, which is on their side of the setup, not something we configured.
Did AI write this?
Yes. The blog version was drafted with AI assistance and then edited and reviewed by us. The facts, numbers and timeline are ours and match the technical postmortem PDF linked in the post, which is the drier, more complete version. Happy to go into any of the technical details here.