> CERN engineers had considered moving to CentOS Stream as a more natural pathway but ultimately they say "the straw that broke the camel's back" to abandon Red Hat Enterprise Linux for their industrial accelerator-control computers was the "-march=x86-64-v2" compiler flag default as "forced obsolescence" for old hardware.
Incompatibility with x86-64-v2 means that CERN is running some older machines, over 15 years old:
> CERN has clarified that their focus with the migration is on their industrial accelerator-control computers while data centers and experimental computing remain on RHEL/AlmaLinux.
For control systems, it’s perfectly reasonable to be using old stuff. Other than power consumption, you aren’t really losing anything, and I think that they famously have some other equipment there that consumes a touch more power than some old servers.
That's a good thing, it means CERN is getting a lot of mileage out of their IT purchases. Buying new machines and deploying them at todays RAM prices? Looks like they are managing their budget responsibly.
In other engineering fields systems are supposed to be supported for at least 60 years. Now that IT stuff has matured, it might be time to hold it to similar standards of longevity.
I'm a Gentoo dev and work at CERN since a while. Of course I jokingly suggested that to my colleagues. I think it could actually work pretty well, though. Many physicists use Gentoo.
We run most things on Alma Linux, but there's also RedHat in the mix.
A good intersection of CERN and Gentoo is EESSI. They use CVMFS, which is a FUSE filesystem used for software distribution from CERN, and Gentoo as part of their software stack. See links below for more info:
Not a Gentoo dev, but also at CERN. I've been using (headless, I'm not going down the GUI toolkit building rabbit-hole again) Gentoo for the last 15 years of development (both at home and at work). Great OS :)
Honest question, what would be different from doing the same with Debian packages? Building packages from source is trivial in Debian. It is literally one command, not much different from installing binary packages. Hosting internal repositories is completely expected and well tested.
Personally I have used Gentoo for many years, though never professionally. Its secure extensions were very well tested and documented back in the day, before SELinux pretty much took over. The kernel build system was terrific.
But never once did I consider setting compiler flags an important feature. The person who knows what flags are best used with an individual package, that -Os is more important than -O3 for this particular loop, is either the packager (who knows the system architecture) or the original developer (who wrote the code). Never the end user.
Debian is comparably a more mature system. The downside is perhaps that its packages are a bit more complex than they should be. Functionality-wise it can do anything you could wish for. But if you are a big institution with a lot of competent people in house, go right ahead. You can't possibly go wrong with a well tested system and the knowledge to improve the bits you need.
There's even a thing called GRS (Gentoo Reference System) - a predefined Gentoo configuration that predictibly generates identical Gentoo installations. It's probably a chore to set up (that is, more than an ordinary Gentoo), but I expect it would make it very easy to maintain hundreds of installations based on that GRS.
I believe NetBSD would be better for these machines. NetBSD supports old systems quite well and have a release cycle that in cases are longer than Debian.
But, CERN may have some proprietary applications on those systems that force them to use a specific setup which is hard to replicate in other distros.
Most software at CERN is open source. But you are right, for things which are quite specific to CERN, like the control systems of the accelerator, there's also some proprietary code. Same for hardware actually, CERN develops open hardware too.
They weren’t using RHEL. The title would be more accurate if it said “RHEL-based Linux”. They used their own build called Scientific Linux for a long time, then switch to CentOS. Then Oracle started aggressively poaching RedHat’s customers with Oracle Linux, which forced RedHat to start locking things down. One side-effect is that the life of CentOS 8 was dramatically shortened, and then RedHat killed the stable released versions of CentOS (9, 10, etc.). CentOS is now a rolling release “Stream” distribution, which nobody actually wants to use in production.
All this drama alienated many people who built their careers on RHEL-based systems, and created (either actual or perceived) instability in the RHEL ecosystem. And at the same time they started deprecating older hardware, some would say far too early. This is the “straw that broke the camel’s back”, as they said in the CERN presentation.
> CERN engineers had considered moving to CentOS Stream as a more natural pathway but ultimately they say "the straw that broke the camel's back" to abandon Red Hat Enterprise Linux for their industrial accelerator-control computers was the "-march=x86-64-v2" compiler flag default as "forced obsolescence" for old hardware.
Incompatibility with x86-64-v2 means that CERN is running some older machines, over 15 years old:
https://en.wikipedia.org/wiki/X86-64#Microarchitecture_level...
> CERN has clarified that their focus with the migration is on their industrial accelerator-control computers while data centers and experimental computing remain on RHEL/AlmaLinux.
For control systems, it’s perfectly reasonable to be using old stuff. Other than power consumption, you aren’t really losing anything, and I think that they famously have some other equipment there that consumes a touch more power than some old servers.
That's a good thing, it means CERN is getting a lot of mileage out of their IT purchases. Buying new machines and deploying them at todays RAM prices? Looks like they are managing their budget responsibly.
In other engineering fields systems are supposed to be supported for at least 60 years. Now that IT stuff has matured, it might be time to hold it to similar standards of longevity.
15 years is not old at all for a lot of applications!
Is 15 years a lot?
In computer terms it certainly is.
What's your point?
They could have used gentoo with their own binary package server. Maximum flexibility, including all compiler flags.
I'm a Gentoo dev and work at CERN since a while. Of course I jokingly suggested that to my colleagues. I think it could actually work pretty well, though. Many physicists use Gentoo.
We run most things on Alma Linux, but there's also RedHat in the mix.
A good intersection of CERN and Gentoo is EESSI. They use CVMFS, which is a FUSE filesystem used for software distribution from CERN, and Gentoo as part of their software stack. See links below for more info:
- https://www.eessi.io/docs/filesystem_layer/
- https://www.eessi.io/docs/compatibility_layer/
Not a Gentoo dev, but also at CERN. I've been using (headless, I'm not going down the GUI toolkit building rabbit-hole again) Gentoo for the last 15 years of development (both at home and at work). Great OS :)
Yeah, I'm really not sure that's a joke; binary packages, optionally combined with a prefix install, seem like a great fit.
Honest question, what would be different from doing the same with Debian packages? Building packages from source is trivial in Debian. It is literally one command, not much different from installing binary packages. Hosting internal repositories is completely expected and well tested.
Personally I have used Gentoo for many years, though never professionally. Its secure extensions were very well tested and documented back in the day, before SELinux pretty much took over. The kernel build system was terrific.
But never once did I consider setting compiler flags an important feature. The person who knows what flags are best used with an individual package, that -Os is more important than -O3 for this particular loop, is either the packager (who knows the system architecture) or the original developer (who wrote the code). Never the end user.
Debian is comparably a more mature system. The downside is perhaps that its packages are a bit more complex than they should be. Functionality-wise it can do anything you could wish for. But if you are a big institution with a lot of competent people in house, go right ahead. You can't possibly go wrong with a well tested system and the knowledge to improve the bits you need.
I had the exact same thought! One guy could maintain their OS. No forced compiler flags, internal binhost, everything else same as what they had.
There's even a thing called GRS (Gentoo Reference System) - a predefined Gentoo configuration that predictibly generates identical Gentoo installations. It's probably a chore to set up (that is, more than an ordinary Gentoo), but I expect it would make it very easy to maintain hundreds of installations based on that GRS.
I believe NetBSD would be better for these machines. NetBSD supports old systems quite well and have a release cycle that in cases are longer than Debian.
But, CERN may have some proprietary applications on those systems that force them to use a specific setup which is hard to replicate in other distros.
Most software at CERN is open source. But you are right, for things which are quite specific to CERN, like the control systems of the accelerator, there's also some proprietary code. Same for hardware actually, CERN develops open hardware too.
https://opensource.web.cern.ch/
Good. Debian is great and Rhel sucks
Good, I wonder how much money they’ve wasted over the years paying for RHEL.
They weren’t using RHEL. The title would be more accurate if it said “RHEL-based Linux”. They used their own build called Scientific Linux for a long time, then switch to CentOS. Then Oracle started aggressively poaching RedHat’s customers with Oracle Linux, which forced RedHat to start locking things down. One side-effect is that the life of CentOS 8 was dramatically shortened, and then RedHat killed the stable released versions of CentOS (9, 10, etc.). CentOS is now a rolling release “Stream” distribution, which nobody actually wants to use in production.
All this drama alienated many people who built their careers on RHEL-based systems, and created (either actual or perceived) instability in the RHEL ecosystem. And at the same time they started deprecating older hardware, some would say far too early. This is the “straw that broke the camel’s back”, as they said in the CERN presentation.
Eh? Didn't they come up with CentOS initially to not have to pay for RHEL?
I was ready irritated by the title but then again, phoronix has been a joke site for at least a few years.
[dead]