This is also why I pay someone to change my oil. It's a simple procedure, but the list of things that can go wrong is a list.

At a small shop I ran our VCS on my workstation (because otherwise we had none) and at the end of a day I rebooted to do some maintenance on the machine, and it didn't come back online until around 1pm the next day. On the plus side we finally got a dedicated server.

That said, you still gotta touch systems.

If you get off the upgrade treadmill, you will find yourself one day facing a CVE issue that have no patches for the versions you use. And then not only do you have to do a security update, you have to fast-track a version upgrade that your code might not work with anymore.

Last time we had this, I got the team to agree to putting one upgrade of OS or libraries per month onto the task list. We even let the assignee choose what they were going to upgrade (a few had no preferences, and so we'd have them work on whatever was most overdue).

We put some caches in place at work recently, to improve perf and give us some protection against partial outages. Already we are seeing services that cannot handle the same workload they could three months ago.

To paraphrase Jim Highsmith: there are no answers here, and we waste enormous amounts of time looking for them. There are only tradeoffs, with their own sets of consequences to be managed.

The tradeoff I prefer is to drill it into people that shit happens. Something is going to be down, and the more things we have, the higher the likelihood that one of them will be down at any given moment. You can put all of your eggs in one basket and have one outage a year, in which everyone not involved in fixing the damned thing might as well go home, or you have a bunch of systems where I can do 80% of my duties if any one of them is offline.

Systems that are never seen to break (or at least stumble) tend to become siloed, insular. Being able to talk about the 'bigger issue' around an outage is often a time when teams finally communicate with each other (unless you have dysfunctional managers, in which case: run)

And if all else fails, there is always some set of tasks you want people to do but they never seem to have the time for. Having a captive audience isn't the worst thing that could happen to you.

> This is also why I pay someone to change my oil. It's a simple procedure, but the list of things that can go wrong is a list.

This used to be me. I motorcycle, but unlike perhaps your typical motorcyclist I don't have a good brain/hand/mind for working on engines. I just tend to do a shitty job, get bored before I'm done, or can't wrap my head around the geometry of something (so I'll almost always turn a clutch adjuster the wrong way).

So I'd take my bike in for everything. But I was noticing that things didn't quite seem actually fixed, so I brought the bike over to MotoGuild, a place where you can rent time on a motorcycle lift and they have all kinds of tools you can use included, and popped the engine cover off.

I found horrifying evidence that I can't trust anybody to work on my bike other than me. The sprocket where the chain connects to the engine was loose to nearly falling off - it should be tightened to a shitload of newtons and secured with a locking washer. The clutch area was chunky with crud, despite me having paid for a clutch cable swap and them assuring me they had cleaned everything to avoid my new clutch cable getting gnarly and gross like the last one did (I'd lubricate it and black nasty would come out the cable cover). The chain wasn't really cleaned at all, they had clearly just run a rag on the outside, but between the links and on the tire side, it was nasty. Therefore, it wasn't actually effectively lubed, either.

Bunch of other random things as well.

I am like the LAST person to ever say "if you wanna get something done right, do it yourself," because I am flush with cash and bereft of discipline and willpower, but holy shit, apparently, if you wanna get something done right...

On anything with two wheels you better be sure that you inspect it and if at all possible work on it yourself. Your life depends on just about every single bolt. Think ground bound helicopter levels of care.

This is why mainframes running COBOL still are a thing.

If it works why muck it up?

People forget how dependent we have become on computerized automation.

> This is also why I pay someone to change my oil. It's a simple procedure, but the list of things that can go wrong is a list.

If you're not going to change your own oil, take it to the dealership or a mechanic that you personally trust (I doubt this exists as mechanics don't really do routine oil changes). Putting the wrong amount of wrong type of engine oil into your engine is the absolute worst thing you can do to your engine.

I encourage everyone to change their own oil. If you run full synthetic (and you should), you will save a fair amount of money. Even if the shop gives you the right weight, and right amount of oil, you're not getting premium oil, and your oil filter is likely the equivalent of an empty tin can at most Tire and Lube type places.

OEM dealerships are the only places I trust to do any kind of fluid change if I'm not doing it myself.

> as mechanics don't really do routine oil changes

What's this now? Every single mechanic I've ever met is more than happy to do oil changes. Even my "BMW guy," who only works on BMWs and regularly works on $100k+ cars and fun projects like performance upgrades is more than happy to change the oil on my decidedly pedesterian BMW (12 year old entry-level 328xi, worth maybe $5-6k tops).

I won't let anyone change my oil because a monkey once severely crossthreaded the drain plug by feeding it on with an impact wrench rather than hand starting it.

I'm fine with that because I just make damn sure they fix it, and if I have to fight at all for it, I take my business elsewhere.

I had a plumber crack the porcelain on the sink when tightening the drain. They told me about it and bought me a new sink. That's my plumber until I move. In another town the same thing happened and they put some silicone on it and didn't tell me. That plumber was never called again except to tell them why.

I failed an emissions test. The repair place near work wanted to replace the fuel injectors, it was going to cost a third of what the vehicle was worth.

The one near my house pointed out that there was chunk of EGR hose missing and the rest were ratty. They could replace them all for $280 (above the mininum required to address a failed test).

It was the hoses. Went from above the limit on two tests to 10% of the limit. I used that mechanic until I moved across town.

Oh, the pain is real. Cross threading is one of my nemeses. My eye still twitches when I see people cut a bolt on Youtube without threading the nut on first to work as a die (as in 'tap and die'). You're gonna cross-thread that thing! AAH.

I worked as a bicycle mechanic for a bit (lots of bolt cutting, more cross threading than I wanted), but also spent a cold winter working with my dad on a vintage car and I discovered that I don't have as much nostalgia for car work as I thought I might. For the same reason my boss pays for licenses (because you can yell at them and make them pay for the repairs) I pay others to do some of my handiwork.

Despite still being able to turn a bolt the right way while upside down and with the bolt facing away from me, I figure I'm almost as likely to get a rude surprise due to my own stupidity as theirs. If, and this is a big If, I've developed a relationship with the vendor, instead of just driving down to the Jiffy Lube where I'm nobody.

Here's a simple trick that will help you to never ever have a cross threaded bolt: turn the bolt the wrong way until it clicks. That's the step of the threads disengaging from each other. Now they're perfectly aligned to mate so you move forward just a 10th of a turn or so and it will immediately engage. If it doesn't there is something wrong and you need to inspect the threads.

In my opinion, the true antidote is reproducible, and ideally immutable, systems. Maybe this bug would’ve been harder since it’s all the way back to the bootloader, but assuming a more minimal bootloader (perhaps EFIstub directly, or systemd-boot,) you could have something like NixOS where you can, at boot time, jump to a previous configuration with all of the old software.

(Works even better for docker or VM images where you have a supervisor that can move around disk images. Can just construct immutable root images and roll back by booting with an older one.)

I'd also add that you should reboot after updates--even if they haven't touched anything that requires a reboot to put in use.

The reason is that there is plenty of configuration information that is only read during startup, and plenty of programs either only used during startup or that do things during startup that they do not do later.

If an upgrade broke any of that you might not find out until the next reboot. It is a whole lot easier to deal with if you find out right away that the update broke booting than if your first reboot is six months and several intervening updates later.

We hit something like that once. An extended power failure took down several servers. When they came back up several of our things were messed up. We eventually tracked it down to some update months earlier had made a change to how network filesystems got mounted, making them show up later in the boot process--which turned out to be late enough to be after some of our stuff that expected the network drives to be available.

This is especially concerning since the entire value proposition of RedHat is stability and meticulously maintained packages

I thought their value proposition was enterprise support for large companies that want to outsource that ?

Well, and they still do a better job than... a lot of companies.

Notably, this is the second time in a few months that this happens. The previous one was a microcode update, though that one was Intel's fault (according to Intel, it happened only when the microcode was loaded by the kernel; they probably had tested only loading it from the firmware).

> according to Intel, it happened only when the microcode was loaded by the kernel; they probably had tested only loading it from the firmware

I followed the Spectre situation quite closely, and never heard anything along those lines. Do you have a source?

Many system vendors pulled the faulty firmware from their own sites, and Dell went so far as to recommend rolling back if you had installed it, so the partner ecosystem didn't want people running the firmware at all.

> > according to Intel, it happened only when the microcode was loaded by the kernel; they probably had tested only loading it from the firmware

> I followed the Spectre situation quite closely, and never heard anything along those lines. Do you have a source?

Sure, it was this one: https://github.com/intel/Intel-Linux-Processor-Microcode-Dat...

"Intel identified an issue when OS loading microcode update revision 0xDC for cpuid 406E3 and 506E3. The microcode update has been reverted to revision 0xD6. This issue does not affect the microcode update when loaded from BIOS."

Ah, I see. This is for a different firmware issue. They have so many these days, it's hard to keep track. :/

Thanks!

Rarely such updates are deployed into production without testing in non-prod envs

But that is part of the resistance to updating. A whole pile of testing must be done again.

But the testing should be automated so that incremental changes are just that incremental. Stop the world massive changes are an anathema as it so much harder to pinpoint the breaking change.

What can you recommend as a balanced linux distro between the constant updates of arch on one hand and mostly very outdated package versions of debian/centos on the other?

You can't start with a wobbly foundation and then build sturdy structures on top, so:

start with Debian stable and add your cutting-edge packages on top, then watch their sources and mailing lists carefully.

There is of course Fedora. The latest is quite cutting edge, but you can choose when to upgrade to the next release. And there are security updates for older releases, so you don't have to immediately upgrade. Still, keeps you in the Red Hat universe.

This is exactly why I settled on Fedora after a few years on Arch, a few on Ubuntu/Debian/etc. Fedora is the best blend I've found. New releases about twice a year but you can skip a release if you want and upgrade every other release (or always stay on release n-1 which some people I know do).

Arch and Fedora follow the same conventions so everything on Fedora is where you expect it from Arch (this also makes much of the great Arch wiki applicable directly).

There will always be a soft spot in my heart for Arch, but Fedora is the perfect balance for me. I can't have my work machine getting broken and forcing me to read forums and mailing lists, etc.

Instead of being afraid of updates you should be able to rollback any problematic updates and be impossible to end up in a broken system because updates interrupted. The only distros offering that are NixOS and GuixSD.

That's a great start, but does not help if the data was also affected. For instance, running a new version might have upgraded a database to a newer format, or added new data with a format that the older version does not understand.

True, schema upgrades are an issue which hasn't been discussed[1]. Maybe some kind of versioning or pre-update backup in case of non-invertible data upgrade could help?

[1]: https://discourse.nixos.org/t/automatic-database-schema-upda...

I've found Ubuntu LTS to be a reasonable balance between stability and "newness," and many applications and libraries test with it.

Obviously, perfect stability and perfectly up-to-date packages are not possible, but I think Ubuntu LTS is a good compromise. I've used Arch (which I enjoy using to improve my understanding of Linux w/their excellent docs), Debian, Fedora, OpenBSD, etc and I always find myself going back to Ubuntu LTS for my work machine.

Lots of projects also put out PPAs with frequent releases for Ubuntu, which are ideal if you need the very latest and greatest. Just realize that you're relinquishing root access to every PPA you add, so be very careful whose PPAs you install, and try to limit them to the extent possible (same applies to any apt repo, obviously).

So you want stable + cutting edge?

Yes. That's the "simultaneously too fast and too slow" that all distros have to grapple with.

You might take a look at openSUSE Leap. I use the rolling update variant Tumbleweed for my laptops. But my feeling is that they are working hard to find the sweet spot between stable and bleeding edge.

This one is actually quite interesting: on a long running server with kernel livepatching, you might not even notice it if it gets fixed before you reboot the next time.