If you had to spend the weekend updating kernels, you might want to look at your overall system architecture. Replacing a node with another one running a newer kernel shouldn't be a stressful or time-consuming task; it's part of the normal progress of the system.

This greatly depends on the organisations size and if you're on-prem vs cloud. If you have a small-medium startup with a single product and a fairly simple architecture in AWS this might make sense. But, if you're a bank or something with 1000's of applications, 100k+ servers on-prem, and a global footprint with availability requirements this is a vastly different story.

These are obviously two extremes but you can see there is tons of stuff in the middle two. The larger folks are the ones that are stressed 24/7 even when they have the tools to do it.

The opposite actually. If you're that size it's insane to not have fault tolerant clusters, on-prem or cloud doesn't change much. Time to invest in it.

Yeah, wishful thinking. I work with them everyday and get to see under the hood. I 100% agree with you on investing in fault tolerant clusters / automation. There is a massive trend happening to modernise legacy infrastructure. Almost all the big companies will have some internal platforms team that are trying to standardise. Just extremely slow and tons of training and education.

I don't think it's the newly build stuff that companies are worried about. It's the old legacy stuff that's on life support that no one wants to modernise or maybe they cannot.

The existence of small and large companies with bad systems architecture isn't an argument for implementing more bad systems architecture. Small or large, it doesn't need to cost more to design stuff well.

For what it's worth -- I agree with you and is why folks are flocking to docker/kubernetes and devops tooling like Terraform. I just think you're missing the scope of it. Say for example, a big box store, they might have 500 locations, and all the sudden they buy a few more companies, and merge all this stuff under a single brand. All the sudden they have tons of systems all over the place, lots of existing platforms that need to talk to each other, and staff that are resistant to change. This isn't "newly designed stuff" it's systems that sort of organically grew over a long time. You're talking lots of different operating systems, networks, front-end/back-end systems, www corperate site, mobile site, rewards sites, all sorts of internal support applications, pos system backends, etc. They probably have 20+ different large database systems and they might not even know all the apps connecting to them. I actually worked with a company like this on a few cloud projects. It was amazing to see the complexity. This is the type of stuff that's running large parts of companies that you interact with on a daily basis.

I guess what I'm getting at, is that sure if they design new things they will follow modern patterns but there is so many things that are not modern. They don't have the time or incentive to just go and rebuild all this stuff. There is zero benefit to them on a bottom line, unless there is some burning fire, a way they can extract more money, or save tons of money. So, they just keep them on life support and run in a keep the lights on mode until something happens. These are the systems all sysadmin's just wish went away and there's many of these types of things all over the place.

Yeah, I agree. Your initial reply to me didn't mention legacy issues, just org size, and I was interpreting it more in the context of a large org building new systems the same unmaintainable way they always have because of inertia / politics / ossified sysadmins / etc.

The problems are around process almost always. But I’ve also seen on occasion some sadly pragmatic reasons for very slow processes such as critical legacy software that a replacement for can’t be found even with 8+ figure checks to vendors waved.

One of the big trends from the 2010s for cloud software was to cloudwash old stacks that really weren’t more than simply validated to not crash and burn on an EC2 instance, which is why the entire cloud native movement exists to differentiate greenfield cloud architecture services from cloudwashed ones.

Things can be pretty frustrating working with different vendors of different applications and competencies. People will be patching log4j issues for years to come, for example, and that’s probably easier to validate in aggregate than entire kernel upgrades for decrepit, unsupported distros like CentOS 5 that I still hear about being used.

If you are starting from scratch, sure.

Real life is more complicated, and even if the organisation willpower and politics are aligned in a way to _want_ to fix it, this takes a long time.

Chastising someone on HN because they own a system that probably wasn’t designed and might not have the power to fix seems at best, a little unfair.

I agree, there are many reasons why bad systems architecture might remain in place due to inertia, lack of resources, organisational politics, etc.

I didn't chastise anyone.

Sorry, that wasn’t addressed to you specifically, just aiming upwards in the thread.

A modernization effort is like adding a new, alien science officer named Spock to your crew. Who has the patience, skill, and charisma to deal with the annoying Vulcan and his cold logic?

I have seen a lot of places, yet not a single one that runs all the latest software everywhere all the time. This is even true for smaller pieces of software like Linux kernel.

How would fault tolerance help with privilege escalation?

Would it not just mean that you have more computers to update in your redundant/tolerant cluster?

In this context it means you can take individual servers offline without taking your entire service down. So you can then update each server (even on production systems) live without requiring a maintenance window.

For bonus points you're also not babysitting manually provisioned servers but instead have your software installs automated. So any failure on a server or OS update isn't seen as a maintenance piece but rather just terminating the old server and letting your pipeline auto-build a new server. This is often referred to as "treating your servers as cattle rather than pets", though not everyone likes that analogy.

In the context of a cluster, fault tolerance allows you to replace nodes without downtime. With automation a kernel update can then be a routine, low effort, low stress task.

Honestly, a kernel update has to be a routine, low effort, low stress task. It's a common event that should be seen as part of the normal operation of the system, not as some exceptional event that means someone has to work on the weekend.

Theoretically it means that you can be running regular node OS updates as a matter of course simply by replacing some percentage of them on a rapid cadence.

Then there isn’t any stress to doing it, it’s routine and automated.

You just made a bunch of assumptions about what their system is and does.

Not really? I said they "might want to look at their overall system architecture". I didn't say "your system needs to be redesigned". Maybe it's the best it can be given their constraints. Hence the word "might".

Support in a multinational, clock in, clock out.

No stress, (relative to what you are doing now).

You can hand off to another team at the end of your shift.

You got HR, perks, pension etc. You just gotta eat a bit of shit :)

Yes there is a ton of downsides too but look, you are a Linux admin you may as well sit back and use them skills for a while so you can de-stress and get your life back.

Not sure where you are based but there is a massive demand for Linux admins in Ireland (and probably Europe)

Build systems that can cope with the loss of nodes and ideally self-heal if you kill a node.

Do the major distros not share and coordinate such high-impact security issues with each other? 1015 is tracked for RedHat bugzilla since 2022-03-17. Can't find any information relating to Gentoo, Arch or NixOS, and no fix for Debian or Ubuntu.

Is there a patch one can apply in the meantime? The RedHat suggested mitigation requires disabling functionality that is heavily depended on.

1st 2 links report cve id not found Redhat & Ubuntu have cookie walls and Debian pages are the only readable without pressing or clicking or agreeing or disagreeing on anything. However none of the sites mention a fix.

What is the fix? Surely they don't release this kind of info without a fix.

The oss-security message has the commit hashes, these links (aside from the NVD, which seems to just lack information for the moment) are is only needed to figure out when their backports hit your distro kernel. The answer to that seems to be “not yet” for all the liked distros, which is weird.

I do not see a mitigation mentioned, but the impact of both of these seems to be limited to users with the ability to install nftables bytecode, so it seems having user namespaces disabled (if you don’t need them) would make this irrelevant?

Yes, this confuses me as well. It would be good to know what kind of fixes the grandparent has applied.

Further down "MayeuIC" wrote 1015 is fixed in latest lts and stable kernels. Idk how to link to particular comments here. I'm on the phone and copy/pasting is a bit difficult.

So bottom line - upgrade kernels to latest of your branch, if you're able.

> anyone have suggestions for my next career move?

I'm keeping my eyes open on circus website careers pages. I reckon I'd have way fewer clowns to deal with if I was an actual clown car driver... :sigh:

Maybe look at Linux kernel livepatching, that should reduce your need to reboot as often.

> anyone have suggestions for my next career move?

https://www.goatops.com/

At least point 1 is solved now.

Infosec

> 15 years as a sysadmin, anyone have suggestions for my next career move?

I made the switch to Technical Product Marketing after 15+ years doing linux sysadmin stuff. This might seem weird at first but all tech companies have complex products that they are trying to sell to a technical audience. Marketing needs technical folks embedded that can translate between the tech stack and marketing speak. You can probably 2x your sysadmin compensation quite easily and offers tons of career growth (developer relations, tons of conference speaker opportunities, become some industry expert, etc).

No idea about your skill set or area of expertise but here's an example from vmware [1]. Just search for "Technical Marketing". The job typically involves doing technical reviews of competitors, something you'll already do when deciding to choose a product as a sysadmin, reviewing internal marketing content to make sure people are telling the truth, doing talks/training, recording demos, interacting with PM/Eng about product releases, testing and writing about new releases, etc. If you like the technical side and don't mind teaching this can be a good transition. You basically leverage all the skills you've built over 15 years and apply them to something else quickly.

The kicker here is that you can just apply to companies where you already use their products and know them inside and out (giving you a massive advantage compared to other people applying). Say, you do tons of AWS stuff, well who better to work with marketing on the technical side then a sysadmin who breaths this stuff everyday, or maybe you're doing stuff on cisco switches [2], or maybe some netapp storage fabric expert [3], same thing. All these companies have technical roles in marketing that want you and it can range from mega corps to cool startups like GitLab [4].

[1] https://careers.vmware.com/main/jobs/R2204162?lang=en-us

[2] https://jobs.cisco.com/jobs/ProjectDetail/Technical-Marketin...

[3] https://jobs.netapp.com/job/Bangalore%2C-Karnataka-Technical...

[4] https://about.gitlab.com/job-families/marketing/technical-ma...

As an industry, we need to stop pretending that computer security is a possibility. Assume everyone on the Internet has full access to any network-connected computer, and arrange your affairs from that assumption.

SRE. Learning to build the automation that makes changing the kernel a change to a config file (Ansible/Salt/Nix/etc.) and pushing a button.

It's because of user namespaces lol this is gonna be the next decade, sorry