But are the contracts actually useful? I wager that to have support from Red Hat is really useful beyond what you may view as "corporate politics".
In terms of actual, material benefits I believe it's simply because transitive "certifications" (hardware, support, continuity, ...) that are implied by Red Hat's corporate presence.
We target Ubuntu and Alpine, but also ship CentOS/Rocky/RHEL/Oracle/Alma and Windows. Even though customers rarely ask for the latter targets, we need to offer those simply because we wouldn't be taken seriously if this major market segment were absent on our website.
RHEL compat per se is irrelevant, it's the 10 years guaranteed maintenance enterprise customers are after, plus a path forward for their third-party software investments only certified on RHEL (because of said guarantees) and internal IT deployment/admin line processes. Despite making the rounds on HN, enterprise and other commercial users give a flying fuck to io_uring, systemd (up until RHEL 7 when they were forced to), namespaces, and other Linux "innovations" which in fact are just annoyances to justify contracts for 30+ years of ongoing maintenance of an age-old POSIX core operating system once developed in a couple of months and praised for its minimalism.
Or perhaps they "don't give a flying fuck"?
This ommission of the negative really grinds my gears, and I couldn't care less about American Idiom when it literally changes the meaning of the words.
> This ommission of the negative really grinds my gears, and I couldn't care less about American Idiom when it literally changes the meaning of the words.
Tough luck? Language is like that, idioms especially. Just get used to the fact that not everybody speaks the same dialect of English as you do, and none of them are objectively the correct one in any sense.
You clearly knew what they meant, so there's no communication barrier here. You're being pedantic for the sake of it.
Sorry for my choice of words, you may be right after all, English isn't my native language either. But of all idioms, yours isn't without irony either ...
https://en.m.wikipedia.org/wiki/List_of_common_misconception...
This specific case has been addressed multiple times by an authority. https://www.merriam-webster.com/words-at-play/could-couldnt-...
Are there time-to-patch metrics for comparison with and without stream as compared with IDK Fedora git?
Rpms/kernel explains: https://src.fedoraproject.org/rpms/kernel :
> The kernel is maintained in a source tree rather than directly in dist-git. The specfile is maintained as a template in the source tree along with a set of build scripts to generate configurations, (S)RPMs, and to populate the dist-git repository.
Here are fedora's kernel patches: https://gitlab.com/cki-project/kernel-ark/-/commits/ark-patc...
RHEL <= 9 has kernel-lt (Long Term) and kernel-ml (mainline) kernel patchsets and packages: https://pkgs.org/search/?q=kernel-lt https://www.google.com/search?q=kernel-lt+kernel-ml
Notable Red Hat Enterprise Linux derivatives: https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux_deriv...
List of commercial products based on Red Hat Enterprise Linux: https://en.wikipedia.org/wiki/List_of_commercial_products_ba...
There used to be a stack of numbered kernel patches in the kernel SRPM Source RPM, but now what is the best way to diff centos-stream-9's main branch with Torvalds/git:main? AFAIU kernel distributors must dist patches because GPLv2; packaging workflows are irrelevant to license terms? https://gitlab.com/redhat/centos-stream/src/kernel/centos-st...
Fedora and many other distros do a lot of valued work, too.
FWICS there are FIPS kernel variants for Ubuntu <= 20.04 LTS (2020) but not 22.04 LTS (2022), and Debian and Ubuntu don't have the selinux policy set that Fedora and RHEL+EPEL have. https://ubuntu.com/kernel
From https://news.ycombinator.com/item?id=36480033 :
> Would it be feasible to sed-replace the RHEL and/or Fedora selinux and container-selinux rulesets for use with other Linux distros?
> "AFAIU only SUSE can run both AppArmor and SELinux?*
> And browsers are running as unconfined in selinux with like all major distros; even on ChromiumOS
Act like you added `systemd-nspawn respawn` to every SysV-init script and correctly formatted the epoch time in the correct column of each of the log files to merge and then logship again.
Conda-forge's GitHub PRs to feedstocks model works, but there are no grub or uboot or selinux-policy-targeted or container-selinux or kernel packages in conda-forge or Alpine linux. https://src.fedoraproject.org/rpms/container-selinux/blob/ra...
> Which distro has the best out-of-the-box output for:?
systemd-analyze security
SystemCallFilter=@basic-io
SystemCallLog=~@basic-ioBecause RHEL, and subsequently it's clones, are a really good server OS. You get updates for 10 years, Red Hat or it's employees maintain a large chunk of the generic-Linux stack, large software support, Cockpit, more modern features when compared to the Debian version at the same time (e.g. dracut, firewalld, systemd, NetworkManager, Podman etc).
Then when you want commercial support, converting to RHEL (or vice versa) it's painless and quick.
Just like McDonald's: it isn't the best meal you can get, whether in quality, nutrition or taste. But you will know what you are getting, even when you appear on the other side of the planet. You would be better off with a meal in a some local restaurant, but that one is unknown to you.
So corporate will get something they know for their people. Most of them have zero interest in exploring better options, just want their job done today and go home.
Took me a couple years to get together a coalition of commercial support customers to apply sufficient pressure that they'd listen when I got the original author of the patch to break out the small words and crayon drawings and explain what they'd done wrong and how to backport it properly.
Happily, they now generally snapshot Fedora's perl builds pretty directly and the current Fedora team have been fantastic to work with as a downstream but it was a spectacular mess at the time and there are plenty of people out there who still haven't forgiven them (I think I try to act -as if- I've forgiven them but I'm not sure I actually have).
I suspect more man-hours were lost to the (now reversed for quite a while thanks to the Fedora team) decision to have their 'perl' package only be half a perl install (and to mass report the resulting problems that decision created to cpan authors without ever sending a single patch) but the original subject was things they botched by accident rather than things they broke deliberately.
Though the uninformed arrogance behind the poor decisions and dismissive attitude to the resulting problems, even when they were impacting RHEL support customers, was very similar in both cases.
Had they responded to realising they'd completely broken a bunch of paying customers' primary revenue generating applications by actually trying to do something about it I would have been happy to file the original mistake under "shit happens." As it is, I don't think 'pretty good track record' applies, I'm afraid.
That ONE time might have been fun, but with Arch, Ubuntu and so on, you get to have such fun times all the time.
I can’t count the number of times some rando Microsoft patch has boned a massive global infrastructure.
For those that do not know, RHEL kernels are a bit of an abomination with both bugfixes and new features backported, just so they can tell to customers that the kernel version is "stable" (which is a bit of clown fiesta as Linux doesn't break backward compat on kernel level anyway) and that was one of the things mis-backported and not even tested, just copied bug from upstream.
.... which half a year later got backported to Centos 6 too...
They also do nasty shit like re-enabling/backporting legacy ciphers to software (like OpenSSH) that removed them or disabled them in code base. So you go thru audit, there are points about not using this and that now-insecure ciphers, but you look and see "hey, this one was disabled by default in <current version of OpenSSH in RHEL>, we're fine", and discover that you're NOT fine because RHEL in their infinite wisdom enabled it by default because some old legacy customers needed it and couldn't be bothered to edit their own configs.
After near-two-decades I want to get off Mr. Red Hat weird ride. Just fucking use Debian
Two alternate kernels will restore most missing drivers, El Repo or the Oracle UEK.
Just because Redhat handles this poorly (like pretty much everything), does not make the model bad.
For example, almost every single security update on Debian, is backported when not handled by upstream. And there is a lot of that.
You lose so much stability, if you track all-new. In fact, you literally spend more time chasing underlying bugs, instead your bugs, and dealing with api changes, if your stack is bleeding edge.
Playing around with distros and bleeding edge is fun until you just want the OS to get out of your way so you can do stuff. Then you regret your decisions.
I don’t run Arch, BTW. Debian.
"In 2014, the CentOS development team still had a distribution with far more marketshare than resources."
[1]https://arstechnica.com/gadgets/2020/12/centos-shifts-from-r...
Lots of redhat there.
"Stability" in the RHEL sense is akin to ossification: you don't change anything, and then you can't change anything, and then you've got problems. The costs aren't necessarily as obvious as pursuing stability via dynamism, and in any case can often either be avoided by limiting the lifespan of the system as a whole or at least punted on to a successor.
For military applications, the trade off is even more intense: you build things hoping that they'll never be used, but knowing that they need to work when called upon. Stuff that'll be on the front line tomorrow can have a very different set of lifecycle guarantees from stuff that's got a planned life of 25 years but which (from experience) you know will probably still be around in 50.
and do main vendors (amd, intel, nvidia, whoever build motherboards) provide guarantees that they continue production of specific version of product for the decade?
> They also demanded RHEL exclusively. It's a great example of how extreme stability
is there evidence that rhel is more stable than debian?
This isn't a knock at Debian in any way and has nothing to do about whether or not it's more stable (in terms of uptime) or not.
Well, yes Debian versions are supported for 3 years, RHEL for 10+.
"Stability" means changes, not crashes (although they can be caused by changes). A piece of software compiled 10 years ago will still run today and the systems can continue to get security updates, without needing to change the rug.
if upgrade path is easy and stable, it is obviously more sustainable to upgrade versions every N years, compared to situation when you got 10yo EOL infra far behind mainline with no clear way to support it in the future.
I've heard this kind of thing before, but it's worth considering that utility, government, and infrastructure software may be under different kinds of constraints than what you're used to.
Testing field upgrades takes years and the rollout often takes a year or more. There are hundreds of different teams involved and every one of them wants to ensure they don't miss the thing that bricks a submarine, satellite, transmission line, or IRS portal. If you can do this half or a quarter as often you can save taxpayers tens of not hundreds of millions of dollars.
And in a surprising number of cases the software is never upgraded for the ~25 year or so lifetime of the hardware because the cost of the upgrade is comparable to the cost of buying new hardware. Sometimes that means some poor development team backports security patches for antique versions of DOS or Unix, but in more cases than you'd think they just airgap the computers and hope for the best.
I worked on a government project where new software was being built on COBOL because they had those devs available. No commercial company in their right mind would build any new applications in COBOL, but government is a different animal.
But given the number of COBOL platforms available that have dwindling numbers of maintainers, someone with that skillset can demand a very decent chunk now, I'm told.
In terms of stability, as others have pointed out all that means is "the vendor won't change it and will keep providing it"
Yes. Certain SKUs can be available for surprisingly long times. When you make that part of the RFP or whatever, you get models with that guarantee.
For a specific use case where this mattered, my team procured a bunch of HPE servers and storage with a 10 year service commitment with a defined resolution time. So parts are pre-positioned and physical repairs will made within 4 hours. Drivers and such are required to be supported for that time as well.
> is there evidence that RHEL is more stable than Debian?
RHEL releases are fully supported (including fixes etc) for 10 years. Debian targets a 5 year lifecycle.
There’s nothing wrong with Debian. But just like an enthusiast who wants to live in the edge wouldn’t be happy with Debian (because it’s too stable!), a F500 wouldn’t be for their own reasons.
Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years (from https://wiki.debian.org/LTS)
RHEL 7 is 10 years, and probably a little longer after their recently announced EL7 lifecycle extension ( from: https://access.redhat.com/support/policy/updates/errata/ )
If you open a support ticket and say "Hi I'm using Debian/Arch/Nix" they will just laugh and close the ticket.
SAP: https://community.sap.com/topics/linux/supported-platforms
PeopleSoft/Oracle RDBMS: https://docs.oracle.com/en/database/oracle/oracle-database/1...
But yielding that point would mean that RH/IBM would have you over a barrel on pricing. Probably why Suse is forcing the issue.
P.S. and Oracle is piling on ...
When you're a part of an international research infrastructure which works on the same stack for a decade and half, being supported by RH for free (because of CERN) and creating tons of software at every layer of this research stack over that time period is one of the bigger reasons, but it's not the only one.
I personally use Debian for my personal computers and small servers elsewhere for 15+ years, but when it comes to infrastructure homogenization amongst high thousands of nodes, it's a different story.
Why, did dependability and enteprise long term stability needs go out of fashion?
It seems more straight forward to be able to look at a change log from the application, lets say httpd, then to go search for cherry picked back-porting changes.
https://downloads.apache.org/httpd/CHANGES_2.4
verse
https://centos.pkgs.org/7/centos-updates-x86_64/httpd-2.4.6-...
I just picked httpd since its a common application and as far as I can tell all the updates are minor release numbers. If you have another example that would be useful to see for consideration as I'm genuinely curious which applications change so much that people are dependent upon back-porting for stability.
Until recently even Xilinx's tools would balk at installation on Centos.
When the corp has spent millions on EDA tools, RHEL's yearly subscription looks like peanuts.
Hey, here’s a comprehensive guide complete with commands, expected outcomes, and side effects of each command for setting up a Windows Domain using an RHEL domain controller: https://access.redhat.com/documentation/en-us/red_hat_single...
Oh, that’s for 7.5? One that’s 15 years old and no longer under standard support? It still works! Wait, but you’re a new customer. Sorry, here’s the one for 9: https://access.redhat.com/documentation/en-us/red_hat_enterp...
It is like this for everything. The documentation is clear, concise, comprehensive, complete, and versioned. This is what being RHEL compatible gets you.
Changing an industry standard isn't impossible, but it will be a paradigm shifting challenge akin to moving mountains.
* with blackjack! and public git repos!
Rocky, Alma, and Oracle cannot do that because they are never going to write the documentation, do the testing, or provide the certifications that make a RHEL release a RHEL release.
The point releases of a RHEL fork are meaningless unless they are identical to RHEL. If you cannot get real RHEL, your better off with CentOS Stream than an incompatible point release.
That's precisely what I'm suggesting they attempt.
> The point releases of a RHEL fork are meaningless unless they are identical to RHEL.
Why? (I'm not trying to be belligerent here — I genuinely don't understand why.)
> If you cannot get real RHEL, your better off with CentOS Stream than an incompatible point release.
I thought CentOS Stream essentially is an incompatible point-release — or rather, RHEL is a point-release off CentOS Stream's rolling release.
Does Red Hat provide that much worse documentation for CentOS Stream than for RHEL?
I guess I just don't understand what is the “secret sauce” that makes CentOS Stream so useless, but RHEL so compelling (or even a rebranded-but-otherwise-identical rebuild of RHEL like CentOS 7).
Documentation isn't sexy or something. I don't know why Linux documentation sucks so much, but Redhat makes it suck must less. It takes lots of effort, money and writer hours to create the kind of documentation that Redhat maintains and businesses love documentation. A HOWTO or a FAQ are not documentation.
I actually prefer man pages and I dropped Linux for my personal project years ago in favor of OpenBSD (sometimes FreeBSD) precisely because their man pages are well maintained and really well written. I still have to interact with Linux sometimes and it's almost always Debian or Redhat. And I gotta say Redhat beats Debian on documentation.
Their incentives favour creating Red-Hat-specific documentation, and this then reduces their users' incentive to improve upstream documentation.
Red Hat has contributed work that made distros more homogeneous (cough systemd cough) and excellent package documentation for that work; but also got flak for that...
The developer edition is now more inconvenient, as you have to reapply once a year.
Technical writing is a field of its own and it's important to recognize that. We could ever be so lucky if such writers were as attracted to free software as some developers are.
because you need to pay competent people to do it. As FOSS is made up of lots of volunteers, lots of documentation isn't up to the standard we would want it to be.
Is the documentation for CentOS Stream that much worse? What does being RHEL-compatible get you over and above being CentOS Stream-compatible?
It is true there are no minor releases of CentOS Stream, but that's due to how it fits in the overall RHEL pipeline. A minor release of RHEL is just an internal fork of Stream at a point in time before release where it essentially becomes "patch only", while Stream will continue on with its developments.
Is it the case that RHEL provides vastly better, more detailed changelogs for minor/bugfix updates (even to the public without an RHEL subscription) and that's what makes so much difference that an exactly-compatible rebuild of RHEL is useful but CentOS Stream isn't?
RHEL is entrenched in that world, perfect for IBM (which also lives off those clients).
I really like Nix so far (just started using it), but damn is the documentation ever shit tier.
When Eelco wrote his PhD thesis 20+ years ago, with the first version of nix, it was just an academic project. Now it's a lot more.
Although nix has a very good community, and plenty of contributor activity, there is way more code contributed than docs.
Additionally, very few people are technical writers in the nix community (there is a lot of expertise tho, but mostly on the code side). Therefore we get today's nonstructured docs.
Finally, RH has WAY more money than the NixOs foundation. RH sells stuff. Nix and NixOs are community projects, relying solely on donations. They probably have people whose job is to write docs.
there is an ongoing, organized effort to help docs tho: https://discourse.nixos.org/t/fundraising-for-the-nix-docume...
In practice, your best bet is just asking around on community channels (https://nixos.org/community/)
That, IMO, is a killer feature. I really miss when consuming docs from commercial providers these days.
https://news.ycombinator.com/item?id=35588297
(Still amazed that no one understands these basic points)
Its almost as bad as people believing that Walmart, Target, etc are closing their stores because of "so much theft" as if shrink budgets weren't a thing. Having a loss prevention department is a misnomer, its a paperwork and accounting department so that the losses can be written off to the IRS. These places have KPI's for everything and probably know the locations where they have a store but a majority of people are ordering online. I'm sure their lease has some notion of financial hardship which "theft" falls into allowing them to exit the contract early.
I may not be understanding your point here, but having a shrink budget and closing due to theft aren't mutually exclusive.
It is possible for theft to exceed the shrink budget and make a store too costly to operate.
My main point was that there is no additional cost, even if you exceed the shrink budget. The losses are all written off because with a loss prevention department you have shown, to the insurance company /or IRS, that you are putting in a reasonable effort to deter theft. Stores don't want anyone to actually stop people for a number of reasons the main one being they are writing the loss off or being reimbursed.
An employee getting harmed, the thief getting harmed, employee's getting summons to court, another customer getting harmed are all more costly than just documenting what was stolen. Even in the cases where they build up evidence so that the amount gets close to grand larceny is just paper work. They already have people and technology deployed to do the work its not a new investment.
I did a bit of digging and while "theft" is listed as one of the items "reduced foot-traffic"[2] seems to be the main culprit. They throw out some fancy percentages but the quoted stats seem to be from 2015 and theft is only 1% of their annual revenue[7]. It would be interesting to see the raw data.
[1] https://www.jacksonville.com/story/news/2015/01/15/after-tak... [2] https://www.al.com/news/2023/04/target-store-closings-2023-s... [3] https://www.cleveland.com/news/2023/04/target-to-close-store... [4] https://www.irs.gov/pub/irs-pdf/p584b.pdf [5] https://www.reuters.com/article/us-wal-mart-stores-theft/wal... (from 2015) [6] https://www.wsaz.com/2022/12/09/walmart-may-close-stores-inc... [7] https://blog.gitnux.com/walmart-theft-statistics/
If you cannot figure out, then you are not working with Linux, despite 30y+ of experience.
Big Companies need certifications, maintenance, support and compatibility is necessary. Specially companies running linux in mainframe, or for instance, delivering embedded devices like storages or medical devices
It seems very silly for people to be getting upset now that a free OS hasn't taken their business considerations to heart. Unless of course you were going for the model of charging everyone enterprise rates for your devices/software/work/etc and pocketing what you would have to normally pay in enterprise licensing because its a "community" OS, which everyone is free to do, but these are the risks that come with that.
[1] https://developers.redhat.com/articles/2022/05/10/access-rhe... [2] https://developers.redhat.com/articles/faqs-no-cost-red-hat-...
We paid for support for RHEV/RHV for a number of years and it was useless. You ended up troubleshooting most of the work yourself and in our case we could have just run Ovirt and skipped the whole inventorying and licensing of cores.
The "having someone to blame" or a support contract baffles me for open source. Why advocate for using open source if executive suite is concerned with downtime? They aren't passing the savings back to you and what did having a contract save you if you have to get director or c-level people on the phone to escalate? Presumably they are getting paid more per hour than a number of other people. You have now paid support, plus Dev/Ops troubleshooting, then getting an executive involved. Not to mention how big does your organization need to be for a vendor to consider "your" issue to be a real issue to them.
The back porting and LTS that most people talk about seems to be out of place for a time in which security is now falling under insurance and you need to be regularly updating anyway. Every security audit has one of these "banner shows this version, oh but wait did you check OVAL, or its a back-ported version" conversations. Wasn't this the point of all the fancy DevOps tools, workflows, containerization, etc. Shouldn't you be able to roll out new packages and OS versions quickly since you have all your automated tests implemented?
The reason why it is still special is because even in 2023, if any commercial enterprise applications and hardware support Linux at all, it is generally only RHEL.
I've been using Linux for nearly 30 years. Admining as a profession for at least a quarter of that. 20 years ago, it made a ton of sense. Today, less so.
The 'stable version but we backport patches' mantra doesn't make any sense today. I can't even describe how many things that have broken that you can't even find an answer for because it was some RH specific patch.
Between Debian, Nix, Arch, and others, I can't figure out why RHEL compat is so desirable. I'll go as far as to say that my Arch boxes have been far more predictable than my RHEL boxes.