glorified checklists.
but checklists work (you might have heard about surgeons leaving medical tools in patients, and checklists eliminating this problem, seemingly the dumbest simplest technology, yet it's very powerful compared to the default of nothing)
of course the quality of answers matter, but that's on the environment (auditors, regulators, industry best practices, trade groups, client expectations), see the absolute total shambles of the state of IT sec in South Korea, the 1000 year old COBOL systems in finance/insurance, the reservation system in air travel, the sorry state of TLS before LetsEncrypt, SMTP servers (how SPF, DMARC, DKIM and ARC and MTA-STS are all needed to signal that you really prefer TLS, really don't want to be impersonated, you really vouch for forwarded stuff, etc).
also there's the BeyondCorp step-up. nobody bothered to really segment the internal network. nowadays the default is zero-trust.
These glorified checklists also backfire all the time. With the surgeon it's indeed simple. With software what I see is that certification and process often lessens quality.
Why you ask? Well if you need certification, changes become expensive. Because there's a huge tail of documentation and processes to be done. This leads to the perverse incentive to change as little as possible, even though you know it's broken.
Boeing knew that they needed to retrain pilots if they changed the 737 MAX airplane too much. So they hacked the hardware, which lead to hacking the software, which lead to omission in the training. All because of the incentive not to change too much lest they need to retrain all pilots.
The problem of course is that the retraining is an either or. Either your plane is sufficiently 737 like or not. If not the costs are huge. Especially since the competition's airplane would not need retraining.
So the risks of these hacks to make it 737-like enough were weighed against huge costs. The costs were basically not being able to do business at all. Since these planes would be cost prohibitive for the cheap domestic airlines that want them.
If the costs were more linear I'm sure this wouldn't have happened. E.g. if you can retrain pilots on only the MCAS system and still not require full recertification.
I don't mean any offense toward you.
Just having an incredibly hard time making any sense of what you wrote.
We just issue an updated defect list to go with the software and our customers needs to work around the bug. Known unfixed bugs are a fact of life in certified software. Update releases are not. Customers pay a premium for this because they, too, would have to go through the same pain and expense on their side.
And all the other software.
[1]: https://usatoday30.usatoday.com/tech/science/space/2006-11-0...
It took me 9 months talking to them about this problem until I managed to reproduce it on just two nodes and half a page of code, and record kernel logs that clearly showed a race condition.
We have received a patched kernel in a few days, and it worked like that for a while. This fix was merged into the official release after almost two years.
After that - only Linux, where we can see and fix stuff. No proprietary code and bureaucracy, no "fast, robust and reliable" operating systems.
See also this comment: https://news.ycombinator.com/item?id=35589690
With a safety-certified system, you do not receive a patch because it violates the safety certification. Of course, you can get a patch and use it but then you're responsible for safety-certifying the entire stack including the closed-source vendor code, and best of luck.
They have incorporated the patch into some QNX 6.5 version, but we had to deliver the product long before they did this.
Thank god it does, so that you cant just yolo it and “move fast and break things” with mission critical infrastructure.
Having a forward-looking industry working in symbiosis with its top-notch quality high-functioning regulatory environment is the ideal state. It's rare. (I would say it's apprecition is academic only today. State-capability (or "state capacity") is getting to be a buzzword now for pundits[0][1][2][3]. But it's not that surprising that these problems seem to be cropping up now, in the Internet era, and not during the Cold War.)
My theory is that having a basic regulatory environment allows for increasing industry-wide quality effectively and quickly. For example the CDC is bad at counting COVID cases[2], but catching blindness causing eye drops with "just" ~70 cases countrywide[4] is a good example of the basic safety net.
Similarly, the whole aviation industry's safety process and context was what allowed the MCAS fuckup to come to light fast.
"Fun fact" regarding MCAS. If I recall correctly Boeing argued that the MCAS malfunction was covered by the runaway stabilizer procedure (checklist!), the astronomically bad UX of MCAS itself is what made pilots confused. (Because it activated for 10 sec every minute or something WTF like that, so they had no idea they need to get the runaway stabilizer checklist.) And that's exactly what you're saying. It wasn't sufficiently 737-like.
The workaround is to add UX to these type similarity checks done by the FAA and other regulatory bodies. And, again, exactly as you have mentioned, the cost-benefit discontinuity led to this bad trade off. And while we can't magically smooth over all of them, but at least (and that's my argument) we have good frameworks to start looking at them, detect, recognize, analyze and workaround them. (And checklists are the level 1 of these tools.)
[0] https://marginalrevolution.com/marginalrevolution/2020/01/wh...
[1] https://www.youtube.com/watch?v=zT-McydDEx4
[2] https://www.slowboring.com/p/the-cdcs-vaccine-data-is-all-wr...
[3] https://en.wikipedia.org/wiki/Why_Nations_Fail (but see also "The Narrow Corridor")
[4] https://www.nbcnews.com/health/health-news/recalled-eyedrops...
Seen verifications for ISO-Something, were incredible efforts invested to get the certificate. Looking at the actual technical side…I’m worried.
Certifications which were influenced by people which follow their personal targets are a way to block competition and innovation. Certifications influenced by people with right mindest could help (e.g. require source access, data access, compatibility) a lot.
What I wonder as user:
What does Red Hat and Canonical actually test when ThinkPads are certified?
Particularly for systems that are installed as appliances in customer data-centers, where you as the vendor are both (a) responsible for the appliance as a whole to continue working but (b) have little / restricted access to continue maintaining it, RHEL helps you sign long multi-year support contracts for said appliance with the customer while keeping your maintenance costs to a minimum (since the cost of hiring more support engineers then flying them out is going to be higher than the RHEL license cost).
And I use RHELlikes (Alma/Rocky) in my homelab just cause its similar to Fedora and Red Hat's documentation is much better than reading random blogs on how to do stuff on Ubuntu/Debian.
I often hit it from search results (mostly for software made/supported by RHEL) and it works fine for Debian
Also, OS upgrades work
For example if you are a software vendor who needs to ship a proprietary kernel module - think something like software to control a lab instrument - only Red Hat has a credible kernel ABI scheme with the required scale (because of the reasons I gave in GP). So every ISV in this situation specifies RHEL and requires that you run your software on a purchased copy of RHEL for support reasons. And usually the ISV software costs many multiples of the cost of a RHEL license, so it's lost in the noise.
RHEL did ABI or kABI breaks only with minor releases. CentOS Stream can do them randomly, just like any other distribution.
That said, the CentOS Kmods SIG generates reports on how the kABI churns in CentOS Stream[1] and RHEL[2].
With minor releases you would read release notes, check your modules and knew, whether you can update or not.
Other distros also don't change kernel version during stable phase so I don't even know what you mean by that. Hell, RHEL kernels have more changes than say a Debian kernel over lifetime of a version.
What Red Hat uniquely does is to backport features into older version of the kernel, so statements like "you need kernel x.y.z for this to work" don't really apply, you need to find whether they backported that feature or not.
We've also experienced some "fun" with how they manage it and versions in general for example:
* Driver bug was backported from newer kernel (they backported "performance improvements" not bugs) in rhel5, so we lost VLANs on our upgrade canary * Same bug was backported to rhel6 with same disastrous result 6 months later * One of servers stopped booting because they changed version of LVM in STABLE DISTRO (think it was rhel6), and new LVM version deprecated/renamed some flag that we used, that made it fail on boot.
So yeah, we're very happy we migrated off it. And in Debian distro upgrade works too!
Clearly you've no idea what kABI means. This article (for Red Hat customers) explains it: https://access.redhat.com/solutions/444773
Edit: A different link which isn't paywalled: https://developers.redhat.com/blog/2018/03/28/analyzing-bina...
I agree with you that there is value in kABI stability, but this comment does nothing to convince others. Do you have a better freely accessible resource?
Not everyone (probably not many too) have experts that will track a kernel bug or know how some obscure interaction works.
If you have knowledgable people in-house the RHEL is essentially useless and WILL actively make your life harder compared to just installing Ubuntu or plain Debian as they like to mess up and segment software that normally would be in same repo... but if you don't, and "just need a software that happens to run on Linux", it's cheaper to pay for few servers than hire top level Linux expert.
When we used it many of the "wait, this doesn't work like in vanilla Linux" could be traced to some RHEL engineer changing stuff to work for their enterprise customers.
Like we had audit fail on too old SSH encryption modes enabled. The encryption modes that were disabled and deprecated in that version of SSH but Redhat brought it back with a patch, presumably to support some legacy environments of their customers.
They also do some utter mess with openssl, like removing a bunch of EC curves and various FIPS-related stuff of dubious security impact, re-adding sha1 support and bunch of other stuff
http://mirror.stream.centos.org/9-stream/BaseOS/source/tree/...
LTS is key because migrating OS and recalibrating everything is expensive as fuck for large organizations. As the amount of manhours needed to migrate everything, retest and retool everything and retain and update all your internal documentation is stupendous.
Also having real support for a product is worth its weight in gold in a corporate environment because it allows you to offload responsibilities when stuff breaks.
If 10% of your customers need this for compliance, then you have to test and certify that your stuff works with something like RHEL - and so that becomes an option, or even the default option for the other 90% of your customers.
It's also interesting to see what happens there in some places over time. The older group that manages on-prem is able to hold the "RedHat only" line. But, other groups decide what goes on it, so everything running there is running in non-Redhat containers or nested VMs. It turns into a bulky, extra hypervisor-like layer.
Your comment made me want to say "but COS is certified too!" however now I'm unsure - a quick Google certainly doesn't bring anything up. Why have Google not done this, is the demand not there? Or have they and they just don't document it prominently?
(Note we don't have any PCI requirements as we offload those to the likes of stripe etc)
https://cloud.google.com/assured-workloads
As to your question of why: maintaining compliance is expensive, especially if you're accustomed to pushing prod code changes hundreds of times per quarter.
Red Hat makes more money because its distro is perceived as more of a standard by the ISV community that certified against it. Red Hat does a masterful job of playing both sides of openness - open copyright, closed trademark. They’re pure open source… but they fork most of the upstream with their own branded project version, and convince companies to certify against the proprietary trademarked offering. Automated upgrades and most documentation are behind a paywall. But they also have some of the best security responsiveness and extended support (up to 10 years?) in the industry.
It’s a great business model, but hard to replicate: open in code, but closed in the direction the money must flow.
IBM also loves this model: “WebSphere” certification over “Java EE”, for example. Hence “OpenShift” certification over Kubernetes, or “RHEL” over Linux.
The companies that do have the scale needed to compete are the cloud vendors who are rolling out various certified cloud instances, and in the case of Microsoft/Azure, AMD-based confidential computing.
It’s about the apps and the datacenters. The hyperscalers are formidable competitors, so it’s all about OpenShift becoming the new multi-cloud operating system across them.
Red Hat's code base is a mix of GPLv2/v3, MIT, BSD, Apache - ie, it spans the entire open source licensing spectrum from protective copyleft (GPL) to non-protective (BSD, Apache, MIT). Red Hat is only obligated to release code to GPL code that it ships to customers, but Red Hat goes above and beyond and ships all code that it worked on or otherwise to customers. There is ZERO value in holding back code/fixes etc. Pushing that upstream empowers the whole global community (yes, including competitors). Not doing so will result in fragmentation of the various open source code bases and downstream products.
Trademarks are not the same as copyright. Trademarks tell the receiver who it represents. It is the identity of the entity. So, that is protected and I think that is very fair.
Disclosure: I work for Red Hat.
That was my immediate thought on reading the GP. HAving been in a few Red Hat meetings to discuss possibly using some software of theirs recently, the meetings were indistinguishable from how a meeting with VMware or Veeam goes. They are a company providing enterprise solutions.
That's not meant negatively, that's just what they are, and if you're in the market for an enterprise solution then you appreciate that these companies exist. You get assurances that what you get works as advertised or the provider spends time and effort making it work as they advertised, and you aren't spending your (limited) engineer talent to do so when it can be spent on areas that aren't easily served by throwing some money at it or the cost benefit ratio is far more in your favor.
> OpenShift trying to take on vSphere, Ceph evolving into an enterprise storage player, etc. These aren’t simple moves.
Except it isn't. One of those meetings I had? It was for Red Hat to tell me that they don't really have a good competitor to vSphere anymore, since the RHV platform is being EOL'd and they're getting out of that game. Sure, you can run a VM in OpenShift now, as long as that VM lives in a container and is part of a k8s cluster. The path to success for virtualization with Red Hat is Kubernetes or bust, I guess.
Yes, that's the plan as far as I can tell. The demand for "vSphere but cheaper" is omnipresent. OpenStack and RHV failed, but OpenShift (and ACM) represents a new opportunity and foundation to take a run at vSphere. It will be hard - folks usually vastly underestimate how much smart engineering has gone into that layer.
this may have been, but is not true
We purchase it because we're required to, but we get absolutely no value for the money.
Experience begs to differ
I'm constantly amazed that people still don't understand why Red Hat makes money, and this article won't enlighten you as it doesn't understand it as well as being full of other incidental mistakes (like their description of CentOS is way off the mark).
Red Hat takes Linux and certifies it against all kinds of government, safety, privacy etc standardards, such as PCI and FIPS and numerous local ones. As a result if you are a government or many large companies you simply cannot download Linux and YOLO it. You are legally required to use the certified software so you take the path of least resistance and buy RHEL. That's it. It's why Red Hat makes piles of money, and Canonical or your SaaS does not. Understand what your big customers are required to have and provide it to them, even though it's boring and expensive to do.