It started on the 20th. When they paused builds again at 2am I was still awake from dealing with this and started migrating to a competitor.
There are about 11hrs between when they confirmed Very Bad reads and notifying customers. But it was ~24 hours after they realized someone had some privilege escalation from their build infra.
My mind is boggled by their response.
If someone is trying to get in, literally anyone can point three or four agents at every attack surface and get your shit locked down. To be just fully under attack for... 48 hours? I can't imagine the situation to just be sitting there and taking it.
I told someone half joking that this seemed like they'd been compromised because it was the only thing that made sense. I wonder why google deleted their whole account earlier this year but all the trouble I've had with them I start to wonder less.
This was an awful write up. I want to know how, over multiple days worth of mitigations, the attacker remained in the system. I want to know exactly, from a detailed technical perspective, what flaws there were, what actions have been taken to fix them, and how you can be positive an attacker won’t gain access in the same manner. This post is so high level that it sounds like railway doesn’t even know the answer to them.
> Forensic investigation evidence indicates the attacker was likely looking for AI and platform API keys, for distilling model weights.
It’s impressive that AI API keys are so valuable that attackers are willing to devote resources to hacking a hosting company just to gain access to them.
Another thing: based the actions they have undertaken, it doesn’t sound like they’ve figured out how they were compromised.
> Why did Railway not notify customers of potentially decrypted keys sooner? ... we wanted to ensure the attacker was ejected from our running infrastructure before telling users to roll their credentials, to prevent the credentials from becoming immediately compromised again.
The way this was handled is messed up to the point it's starting to sound hilarious, they could've easily let people know ASAP that the keys were compromised so they can disable them early and Railway could temporarily block them from updating secrets to keep them secure all around in the meantime
The timeline is wild.
It started on the 20th. When they paused builds again at 2am I was still awake from dealing with this and started migrating to a competitor.
There are about 11hrs between when they confirmed Very Bad reads and notifying customers. But it was ~24 hours after they realized someone had some privilege escalation from their build infra.
My mind is boggled by their response.
If someone is trying to get in, literally anyone can point three or four agents at every attack surface and get your shit locked down. To be just fully under attack for... 48 hours? I can't imagine the situation to just be sitting there and taking it.
I told someone half joking that this seemed like they'd been compromised because it was the only thing that made sense. I wonder why google deleted their whole account earlier this year but all the trouble I've had with them I start to wonder less.
This was an awful write up. I want to know how, over multiple days worth of mitigations, the attacker remained in the system. I want to know exactly, from a detailed technical perspective, what flaws there were, what actions have been taken to fix them, and how you can be positive an attacker won’t gain access in the same manner. This post is so high level that it sounds like railway doesn’t even know the answer to them.
That article's tl;dr seems to be: "we recommend you migrate off of Railway". Happy to oblige, already taken care of.
Just curious, where have you migrated to? Plenty of competitors.
For my single simple service, EC2 for now as I need it running reliably and don't have time to mess around with more optimization.
TBD where it lands in the future. Railway was supposed to keep this simple and less messy, but you get what you pay for, as always.
> Forensic investigation evidence indicates the attacker was likely looking for AI and platform API keys, for distilling model weights.
It’s impressive that AI API keys are so valuable that attackers are willing to devote resources to hacking a hosting company just to gain access to them.
Another thing: based the actions they have undertaken, it doesn’t sound like they’ve figured out how they were compromised.
> Why did Railway not notify customers of potentially decrypted keys sooner? ... we wanted to ensure the attacker was ejected from our running infrastructure before telling users to roll their credentials, to prevent the credentials from becoming immediately compromised again.
The way this was handled is messed up to the point it's starting to sound hilarious, they could've easily let people know ASAP that the keys were compromised so they can disable them early and Railway could temporarily block them from updating secrets to keep them secure all around in the meantime
What an absolute mess. Nothing but stress with railway over the past year.