If this was being probed and/or exploited on October 17th, this throws shade on the PHP team for throwing shade on Chainguard in their public announcement.
Essentially, the PHP team is deflecting and abdicating their responsibilities in all this mess they created. "Chainguard broke embargo", "support us with money", "we are just volunteers", "we had knowledge years ago but no POC because we wanted to ship features and not security fixes". All just deflections.
This fiasco shows me that I would never use PHP for any project I'm part of. The PHP team royally messed up, and they are deflecting the blame left, right, up, and down.
PHP team has always been like this. Instead of creating better software they have always tried to add features with no consideration of backward compatibility.
I don’t know why everyone is so hung up on this patch being the only way to fix the situation. The way that Internet services are attacked is by crafting malicious payloads. Having a proxy in front of your application server is a very smart idea, especially if it can do one job: decode input, and re-encode it in a safe way. In a couple hours, I put together a proxy using Claude, which completely eliminates this php2shell attack and many other others, that might possibly be discovered in the near future. Among other things it just decodes and re-encodes multipart-form-data payloads with its own random sane delimiters rather than attacker-chosen ones.
We use it ourselves in our deployments, and yesterday I released it free under MIT license, for anyone who is unwilling or unable to patch their PHP (which is probably millions of servers right now):
While I get that your intentions are good, if your proxy has a vulnerability, it can be exploited. Many people use nginx as a proxy, and it has its own vulnerabilities that have been fixed over the years. You may have solved one problem while opening up another. Even those fronting services have their own vulnerabilities that have to be fixed.
If a system is accessible by someone else, it is inherently insecure and needs to be hardened.
MediaWiki is the last thing that prevents me never dealing (directly) with it again. And it's hard to imagine a drop-in, migrateable replacement for larger wikis. T_T
Who remembers how careless, reckless, and blithe Rasmus Lerdorf was with the PHP 5.3.7 release he didn't bother to test, because running tests was too much of a hassle, because there were already so many test failures that wading through them all to see if there were any new ones was just too much to ask of him, the leader of the widely used project, in charge of cutting releases?
>5.3.7 upgrade warning: [22-Aug-2011] Due to unfortunate issues with 5.3.7 (see bug#55439) users should postpone upgrading until 5.3.8 is released (expected in a few days).
No seriously, he's literally as careless as he claims to be (when he says that repeatedly, you should believe him!), and his lack of giving a shit about things like tests and encryption and security that are extremely important has caused actual serious security problems, like breaking crypt() by checking in sloppy buggy code that would have caused a unit test to fail, but without bothering to run the unit tests (because so many of them failed anyway, so who cares??), and then MAKING A RELEASE of PHP 5.3.7 with, OF ALL THINGS, a broken untested crypt()!
Does anyone think that's just his sense of humor, a self deprecating joke, breaking then releasing crypt() without testing, that's funny in some context? What context would that be? Do they just laugh and shrug it off with "lol let Rasmus be Rasmus! ¯\_(ツ)_/¯ "?
>You can see the code coverage, test case failures, Valgrind reports and more for each branch.
>The crypt change did trigger a test to fail, we just went a bit too fast with the release and didn't notice the failure. This is mostly because we have too many test failures which is primarily caused by us adding tests for bug reports before actually fixing the bug. I still like the practice of adding test cases for bugs and then working towards making the tests pass, however for some of these non-critical bugs that are taking a while to change we should probably switch them to XFAIL (expected fail) so they don't clutter up the test failure output and thus making it harder to spot new failures like this crypt one.
I wish people who report "exploitation attempts" would be very clear about what they actually observed.
Something that would lead to an uncontrolled memory corruption, potentially leading to a crash, or something that indicates a real, code-executing exploit? Those two kinds events are of very different significance. I can't read from Amazon's announcement which one they think they saw.
There's also a tweet by Matthew Prince that says Cloudflare saw thousands of attempts pre-announcement, https://x.com/eastdakota/status/2113079352682570051, then tens of thousands post announcement, and now millions.
Unfortunately, if you read the announcement as referring to 14:00 UTC, that's several hours after Chainguard published their images. I don't know if it's plausible that people saw the Chainguard builds, understood the issue from those and built exploits, but a precise timeline would help understand better.
https://x.com/dok2001/status/2112811965735202970 was posted at 7:43 AM UTC on October 21, 2026, and Chainguard (according to the PHP Foundation blog post) shipped at 7:41 UTC. One of the big reasons Chainguard broke the embargo was because Cloudflare had been seeing attacks, IIRC.
Yeah — I have log entries I believe show LLM-bot-driven exploit attempts — POSTs to a variety of non-application URLs, blocked on the front end proxy by rules I added that prevent multipart form POSTs -- from an hour before the patch was released.
And an odd little cluster of potential attempts it caught just after 9am GMT. Again I can't see any reason why I would be getting multipart form posts to those URLs, it's definitely an exploit, but it's possible there's another older exploit with that pattern. They aren't WP-type URLs though.
The Cloudflare one refers to the time period after the leak where people had already made PoCs. Amazon’s detection is days earlier which is quite concerning.
So either there was another (but non-public) leak or someone else discovered the exploit almost at the same time as it was about to be published. I guess the latter is much less likely because it would be a major coincidence.
With everybody using LLMs there are no coincidences. Popular LLM platforms are like Oracles that help you identify what other people have been working on. "Find a bug in PHP" might have been useless s few months ago, but once somebody used it to develop a full exploit the LLM will start to regurgitate that information.
Frontier models absolutely have access to current information. I just asked Claude about my fantasy football line up. It knew current injury reports, line ups and stats. Google search AI had even more information than that.
However, I’ve done some original research regarding an area that the internet has wrong and that doesn’t seem to have back propagated yet.
> Not my understanding of how the LLMs currently work. There’s a cutoff date and no new info after that date is included in training.
You are almost certainly correct in terms of how the public models work. I guess it's possible that the labs have some tools connected for more recent stuff like this, but it seems unlikely.
The attacker may have known about the exploit for a long time and were saving it, but upon finding out that it was about to be disclosed, they decided it was now or never.
You're asking: How could a hacker - with an exploit that potentially allows him/her to break into half the computers on the internet - know some secret information that is stored on a computer on the internet?
It could also just have been a run-of-the-mill exploit in WordPress.
A leak is a much simpler explanation than an independent discovery who then also managed to discover before publication that someone else has also discovered it.
I’ve seen probing for it in server logs but not actual exploiting of it. I think the botnets added it to their vulnerability scanners but I have yet to see any examples of an actual working exploit. There are hundreds of vibe coded attempts on github but none of them work to the best of my knowledge. Would love someone to prove me wrong here.
Also this post from Amazon is embarrassing. They knew about it from the 17th yet failed to meet the embargo timeline and patch Amazon Linux (their patch is due to be released today the 22nd of Oct).
I don’t know when various parts of Amazon learned about it, but I don’t think this post means they knew about it on Oct 17. I think they looked at past logs from their honeypots and found a probe for php2shell happened on Oct 17.
What makes you think they knew about it from the 17th? "...dating to Oct. 17, 2026" means that's when it happened. It doesn't mean they identified it on the 17th.
Do they differentiate between attempts and successful infiltration? I’m trying to understand whether this exploit is easy to use or not. There was an article here from yesterday that argued the exploit was not easy to use and needed LFI
It's pretty telling that the attacks started after the vulnerability was found by researchers and PHP insiders notified. Who really believes that a malicious entity found the exact same exploit chain at the same time fully independently?
The PHP foundation needs to give a better answer for why they were asleep at the wheel for 2+ months while insiders were sharing/selling the info.
So Amazon knew about this exploit and that it was actively being used several days before the release of any patches. Despite this they still took 18 hours AFTER the release of the patches to provide updated version of their Amazon Linux packages?
If this is only a retrospective find then they should make it more clear, from the blog it reads to me as promoting their MadPot tool for detection pre CVE disclosure:
"Subsequently, from 2026-10-21 0200 UTC to 2026-10-21 0800 UTC, Amazon Threat Intelligence identified additional exploitation attempts of PHP2Shell targeting MadPot from additional anonymized infrastructure." (i.e almost 12 hours before the release of the patch)
"Invest in detections that don’t rely on a signature for a specific CVE. The first exploitation will often come before those signatures exist."
"We analyze over 200 billion events per minute and integrate threat intelligence from our MadPot honeypot network to identify and block known exploitation attempts in real time. Our honeypot system provides us with early detection of exploitation attempts, enabling rapid response and threat analysis."
My reading here is that they've gone through their logs retrospectively, because you can see these attempts to exploit pretty clearly in an observability platform, and the 17th is only five days ago, so you can assume they've got a lot of fully detailed logs over that period.
> Why haven’t they fixed Amazon Linux then? Same as none of their AMIs for bottlerocket have emSign revocations.
Last I checked, Amazon has also not patched their Lightsail blueprints for WordPress, or even communicated about the vulnerability with those customers at all, which is arguably way worse, because
(a) by design, all-in-one blueprints aren't really intended to use system PHP or have system PHP updated in a meaningful way; and
(b) hosting static sites such as WordPress is 90% of the selling point of Lightsail as a product
That's pretty awful. I do wonder if they've bitten off more than they can chew with all the ancillary stuff to their core IaaS platform. That seems to be the only well maintained bit of their estate, well EC2/VPC/S3/R53 etc.
It's a upstream targeted fix that was made available significantly faster in packages/images maintained by volunteers, it's not some untested downstream patch. Customers at this point would have likely taken their applications offline over this issue so it would have been difficult to make the situation any worse.
Given the sheer number of PHP applications out there, it's impossible to conclude with certainty that this targeted fix will have no regressive impact without significant testing. How much testing needs to be done is debatable, but I'm pretty sure the answer isn't zero.
Someone with a single gigabit internet connection can scan the entire IPv4 address space in less than a day. When you have public RCE vulnerabilities like this, your options are "apply the patch", "take the system offline until you can apply the patch" or "get pwned".
Mitigations and patches are in the same category, so now you're applying mitigations without testing. What if your new WAF detection signature has false positives etc.?
The difference is that a mitigation is customer-applied, not vendor-supplied. Like it or not, blame shifts when a vendor supplies a fix that's insufficiently tested. The CrowdStrike BSOD incident from 2024 comes to mind.
As for WAF rules, those can usually be flipped on and off with a single switch.
Being able to roll it back isn't really a difference, you can revert to the previous version of a package too.
The trouble is if the WAF change e.g. messes up a redirect or turns a success response into an error response or vice versa, which confuses some other system into giving multiple users the same session cookie and gives them access to one another's accounts or writes corrupt entries to the database or whatever it is. You can revert the rule but that doesn't revert the state changes that have already happened.
There is inherently a trade off between keeping the system online, keeping the system patched and testing the patch before applying it.
> Being able to roll it back isn't really a difference, you can revert to the previous version of a package too.
Sometimes. In a world in which customers are increasingly adopting immutable infrastructure patterns and (contrary to best practice) using Launch Templates with symbolic references to `latest` versions of AMIs, this isn't always easy.
> There is inherently a trade off between keeping the system online, keeping the system patched and testing the patch before applying it.
I don't think you're being understanding or empathetic here.
Any vendor that delivers updated packages without thorough regression testing that ends up causing an incident for their customers as a result will get a thrashing from the community. Remember the CrowdStrike BSOD incident in 2024?
Maybe Ubuntu and others are cutting corners to deliver more quickly. It's their risk to take, but it's still a risk.
Unfortunately people rarely follow best practices.
Customers frequently do risky things like using launch templates that refer to the latest AMI. AMI changes, Auto Scaling launches a new/replacement instance in the fleet without adequate testing, *boom*.
All these tech companies are school children comparing how much better their bike is than the others, but they've all got the exact same bike, and that bike is a massive vulnerability that they did nothing to fix, but they're all bragging about how they saw it first or identified it first.
To the person at amazon that thought this was a good press release to put out, I know this is what you were hoping for, so here it is, "wow, so cool". Not sure why, but AI really has everyone running around with desperate loser aura.
Maybe next time, do better, actually patch some things so your customers are covered?
If this was being probed and/or exploited on October 17th, this throws shade on the PHP team for throwing shade on Chainguard in their public announcement.
Essentially, the PHP team is deflecting and abdicating their responsibilities in all this mess they created. "Chainguard broke embargo", "support us with money", "we are just volunteers", "we had knowledge years ago but no POC because we wanted to ship features and not security fixes". All just deflections.
This fiasco shows me that I would never use PHP for any project I'm part of. The PHP team royally messed up, and they are deflecting the blame left, right, up, and down.
PHP team has always been like this. Instead of creating better software they have always tried to add features with no consideration of backward compatibility.
No serious project should use PHP.
I don’t know why everyone is so hung up on this patch being the only way to fix the situation. The way that Internet services are attacked is by crafting malicious payloads. Having a proxy in front of your application server is a very smart idea, especially if it can do one job: decode input, and re-encode it in a safe way. In a couple hours, I put together a proxy using Claude, which completely eliminates this php2shell attack and many other others, that might possibly be discovered in the near future. Among other things it just decodes and re-encodes multipart-form-data payloads with its own random sane delimiters rather than attacker-chosen ones.
We use it ourselves in our deployments, and yesterday I released it free under MIT license, for anyone who is unwilling or unable to patch their PHP (which is probably millions of servers right now):
https://github.com/Qbix/proxy
While I get that your intentions are good, if your proxy has a vulnerability, it can be exploited. Many people use nginx as a proxy, and it has its own vulnerabilities that have been fixed over the years. You may have solved one problem while opening up another. Even those fronting services have their own vulnerabilities that have to be fixed.
If a system is accessible by someone else, it is inherently insecure and needs to be hardened.
I would be very careful with a vibe coded internet proxy server handling untrusted input especially written in C
MediaWiki is the last thing that prevents me never dealing (directly) with it again. And it's hard to imagine a drop-in, migrateable replacement for larger wikis. T_T
Who remembers how careless, reckless, and blithe Rasmus Lerdorf was with the PHP 5.3.7 release he didn't bother to test, because running tests was too much of a hassle, because there were already so many test failures that wading through them all to see if there were any new ones was just too much to ask of him, the leader of the widely used project, in charge of cutting releases?
>5.3.7 upgrade warning: [22-Aug-2011] Due to unfortunate issues with 5.3.7 (see bug#55439) users should postpone upgrading until 5.3.8 is released (expected in a few days).
No seriously, he's literally as careless as he claims to be (when he says that repeatedly, you should believe him!), and his lack of giving a shit about things like tests and encryption and security that are extremely important has caused actual serious security problems, like breaking crypt() by checking in sloppy buggy code that would have caused a unit test to fail, but without bothering to run the unit tests (because so many of them failed anyway, so who cares??), and then MAKING A RELEASE of PHP 5.3.7 with, OF ALL THINGS, a broken untested crypt()!
http://i.imgur.com/cAvSr.jpg
Does anyone think that's just his sense of humor, a self deprecating joke, breaking then releasing crypt() without testing, that's funny in some context? What context would that be? Do they just laugh and shrug it off with "lol let Rasmus be Rasmus! ¯\_(ツ)_/¯ "?
https://www.reddit.com/r/programming/comments/jsudd/you_see_...
>r314434 (rasmus): Make static analyzers happy
>r315218 (stas): Unbreak crypt() (fix bug #55439) # If you want to remove static analyser messages, be my guest, but please run unit tests after
https://web.archive.org/web/20140818115440/http://svn.php.ne...
https://web.archive.org/web/20111014213651/https://plus.goog...
>Rasmus Lerdorf
>+Lorenz H.-S. We do. See http://gcov.php.net
>You can see the code coverage, test case failures, Valgrind reports and more for each branch.
>The crypt change did trigger a test to fail, we just went a bit too fast with the release and didn't notice the failure. This is mostly because we have too many test failures which is primarily caused by us adding tests for bug reports before actually fixing the bug. I still like the practice of adding test cases for bugs and then working towards making the tests pass, however for some of these non-critical bugs that are taking a while to change we should probably switch them to XFAIL (expected fail) so they don't clutter up the test failure output and thus making it harder to spot new failures like this crypt one.
I wish people who report "exploitation attempts" would be very clear about what they actually observed.
Something that would lead to an uncontrolled memory corruption, potentially leading to a crash, or something that indicates a real, code-executing exploit? Those two kinds events are of very different significance. I can't read from Amazon's announcement which one they think they saw.
There's also a tweet by Matthew Prince that says Cloudflare saw thousands of attempts pre-announcement, https://x.com/eastdakota/status/2113079352682570051, then tens of thousands post announcement, and now millions.
Unfortunately, if you read the announcement as referring to 14:00 UTC, that's several hours after Chainguard published their images. I don't know if it's plausible that people saw the Chainguard builds, understood the issue from those and built exploits, but a precise timeline would help understand better.
https://x.com/dok2001/status/2112811965735202970 was posted at 7:43 AM UTC on October 21, 2026, and Chainguard (according to the PHP Foundation blog post) shipped at 7:41 UTC. One of the big reasons Chainguard broke the embargo was because Cloudflare had been seeing attacks, IIRC.
Yeah — I have log entries I believe show LLM-bot-driven exploit attempts — POSTs to a variety of non-application URLs, blocked on the front end proxy by rules I added that prevent multipart form POSTs -- from an hour before the patch was released.
And an odd little cluster of potential attempts it caught just after 9am GMT. Again I can't see any reason why I would be getting multipart form posts to those URLs, it's definitely an exploit, but it's possible there's another older exploit with that pattern. They aren't WP-type URLs though.
The Cloudflare one refers to the time period after the leak where people had already made PoCs. Amazon’s detection is days earlier which is quite concerning.
So either there was another (but non-public) leak or someone else discovered the exploit almost at the same time as it was about to be published. I guess the latter is much less likely because it would be a major coincidence.
With everybody using LLMs there are no coincidences. Popular LLM platforms are like Oracles that help you identify what other people have been working on. "Find a bug in PHP" might have been useless s few months ago, but once somebody used it to develop a full exploit the LLM will start to regurgitate that information.
> once somebody used it to develop a full exploit the LLM will start to regurgitate that information
Sure about that?
Not my understanding of how the LLMs currently work. There’s a cutoff date and no new info after that date is included in training.
Info to the contrary welcome.
Frontier models absolutely have access to current information. I just asked Claude about my fantasy football line up. It knew current injury reports, line ups and stats. Google search AI had even more information than that.
However, I’ve done some original research regarding an area that the internet has wrong and that doesn’t seem to have back propagated yet.
> Not my understanding of how the LLMs currently work. There’s a cutoff date and no new info after that date is included in training.
You are almost certainly correct in terms of how the public models work. I guess it's possible that the labs have some tools connected for more recent stuff like this, but it seems unlikely.
The attacker may have known about the exploit for a long time and were saving it, but upon finding out that it was about to be disclosed, they decided it was now or never.
How could he have known on Oct 17 that it was about to be disclosed? The dates only leaked on Oct 20.
You're asking: How could a hacker - with an exploit that potentially allows him/her to break into half the computers on the internet - know some secret information that is stored on a computer on the internet?
It could also just have been a run-of-the-mill exploit in WordPress.
A leak is a much simpler explanation than an independent discovery who then also managed to discover before publication that someone else has also discovered it.
I’ve seen probing for it in server logs but not actual exploiting of it. I think the botnets added it to their vulnerability scanners but I have yet to see any examples of an actual working exploit. There are hundreds of vibe coded attempts on github but none of them work to the best of my knowledge. Would love someone to prove me wrong here.
Also this post from Amazon is embarrassing. They knew about it from the 17th yet failed to meet the embargo timeline and patch Amazon Linux (their patch is due to be released today the 22nd of Oct).
I don’t know when various parts of Amazon learned about it, but I don’t think this post means they knew about it on Oct 17. I think they looked at past logs from their honeypots and found a probe for php2shell happened on Oct 17.
What makes you think they knew about it from the 17th? "...dating to Oct. 17, 2026" means that's when it happened. It doesn't mean they identified it on the 17th.
Do they differentiate between attempts and successful infiltration? I’m trying to understand whether this exploit is easy to use or not. There was an article here from yesterday that argued the exploit was not easy to use and needed LFI
I've been having trouble exploiting it from first principles even in lab conditions.
It's pretty telling that the attacks started after the vulnerability was found by researchers and PHP insiders notified. Who really believes that a malicious entity found the exact same exploit chain at the same time fully independently?
The PHP foundation needs to give a better answer for why they were asleep at the wheel for 2+ months while insiders were sharing/selling the info.
Well, according to OpenAI:s blog post, they led everything, other than the fix and release. I guess leader takes the responsibility?
So Amazon knew about this exploit and that it was actively being used several days before the release of any patches. Despite this they still took 18 hours AFTER the release of the patches to provide updated version of their Amazon Linux packages?
https://github.com/amazonlinux/amazon-linux-2023/issues/1148
I don't think this blog post is sending the message AWS intended...
What makes you think they knew about it on the 17th? Scanning old logs is not the same as understanding them as they're generated.
If this is only a retrospective find then they should make it more clear, from the blog it reads to me as promoting their MadPot tool for detection pre CVE disclosure:
"Subsequently, from 2026-10-21 0200 UTC to 2026-10-21 0800 UTC, Amazon Threat Intelligence identified additional exploitation attempts of PHP2Shell targeting MadPot from additional anonymized infrastructure." (i.e almost 12 hours before the release of the patch)
"Invest in detections that don’t rely on a signature for a specific CVE. The first exploitation will often come before those signatures exist."
"We analyze over 200 billion events per minute and integrate threat intelligence from our MadPot honeypot network to identify and block known exploitation attempts in real time. Our honeypot system provides us with early detection of exploitation attempts, enabling rapid response and threat analysis."
My reading here is that they've gone through their logs retrospectively, because you can see these attempts to exploit pretty clearly in an observability platform, and the 17th is only five days ago, so you can assume they've got a lot of fully detailed logs over that period.
Take a peek at my detailed full-ish timeline of every actor around the embargo and leak, just added amazon's reporting in there as well:
https://news.ycombinator.com/item?id=50190603
What did crowdsec see? Anybody know? Php exploits should be on a fast track of alerting, given the wide installation base.
Crowdsec spotted more than 300 IPs listed in 24h and the first one were spotted prior the patch release. source : https://www.reddit.com/r/CrowdSec/comments/1xci24u/php2shell...
Why haven’t they fixed Amazon Linux then?
Same as none of their AMIs for bottlerocket have emSign revocations.
Absolutely shit company who puts their customers at risk.
> Why haven’t they fixed Amazon Linux then? Same as none of their AMIs for bottlerocket have emSign revocations.
Last I checked, Amazon has also not patched their Lightsail blueprints for WordPress, or even communicated about the vulnerability with those customers at all, which is arguably way worse, because
(a) by design, all-in-one blueprints aren't really intended to use system PHP or have system PHP updated in a meaningful way; and
(b) hosting static sites such as WordPress is 90% of the selling point of Lightsail as a product
That's pretty awful. I do wonder if they've bitten off more than they can chew with all the ancillary stuff to their core IaaS platform. That seems to be the only well maintained bit of their estate, well EC2/VPC/S3/R53 etc.
-1: if the fix breaks customers in other ways, they're screwed as well. Damned if you do, damned if you don't...
It's a upstream targeted fix that was made available significantly faster in packages/images maintained by volunteers, it's not some untested downstream patch. Customers at this point would have likely taken their applications offline over this issue so it would have been difficult to make the situation any worse.
Given the sheer number of PHP applications out there, it's impossible to conclude with certainty that this targeted fix will have no regressive impact without significant testing. How much testing needs to be done is debatable, but I'm pretty sure the answer isn't zero.
Someone with a single gigabit internet connection can scan the entire IPv4 address space in less than a day. When you have public RCE vulnerabilities like this, your options are "apply the patch", "take the system offline until you can apply the patch" or "get pwned".
The list of options is larger than that. There are mitigations that can be quickly applied. See, e.g., https://ma.ttias.be/php-cve-2026-109618-early-mitigation/ and https://aws.amazon.com/blogs/security/using-aws-waf-to-apply...
Meanwhile, WAFs are being continuously updated with detection signatures.
Mitigations and patches are in the same category, so now you're applying mitigations without testing. What if your new WAF detection signature has false positives etc.?
The difference is that a mitigation is customer-applied, not vendor-supplied. Like it or not, blame shifts when a vendor supplies a fix that's insufficiently tested. The CrowdStrike BSOD incident from 2024 comes to mind.
As for WAF rules, those can usually be flipped on and off with a single switch.
Being able to roll it back isn't really a difference, you can revert to the previous version of a package too.
The trouble is if the WAF change e.g. messes up a redirect or turns a success response into an error response or vice versa, which confuses some other system into giving multiple users the same session cookie and gives them access to one another's accounts or writes corrupt entries to the database or whatever it is. You can revert the rule but that doesn't revert the state changes that have already happened.
There is inherently a trade off between keeping the system online, keeping the system patched and testing the patch before applying it.
> Being able to roll it back isn't really a difference, you can revert to the previous version of a package too.
Sometimes. In a world in which customers are increasingly adopting immutable infrastructure patterns and (contrary to best practice) using Launch Templates with symbolic references to `latest` versions of AMIs, this isn't always easy.
> There is inherently a trade off between keeping the system online, keeping the system patched and testing the patch before applying it.
On this we agree 100%.
Nearly every other vendor has pushed fixes for it, including where they pull their upstream stuff from. There's no excuse.
I don't think you're being understanding or empathetic here.
Any vendor that delivers updated packages without thorough regression testing that ends up causing an incident for their customers as a result will get a thrashing from the community. Remember the CrowdStrike BSOD incident in 2024?
Maybe Ubuntu and others are cutting corners to deliver more quickly. It's their risk to take, but it's still a risk.
Every vendor is winging it there. Even RHEL. Ubuntu ships shit half the time as well even on LTS.
If you need to test it you should pin packages.
> If you need to test it you should pin packages.
Unfortunately people rarely follow best practices.
Customers frequently do risky things like using launch templates that refer to the latest AMI. AMI changes, Auto Scaling launches a new/replacement instance in the fleet without adequate testing, *boom*.
All these tech companies are school children comparing how much better their bike is than the others, but they've all got the exact same bike, and that bike is a massive vulnerability that they did nothing to fix, but they're all bragging about how they saw it first or identified it first.
To the person at amazon that thought this was a good press release to put out, I know this is what you were hoping for, so here it is, "wow, so cool". Not sure why, but AI really has everyone running around with desperate loser aura.
Maybe next time, do better, actually patch some things so your customers are covered?