> Generally speaking I think people need to be more comfortable with build packages for their operating system of choice and install applications that way. The it's mostly configuration that needs
why, it’s 2025, docker / container makes life so easy
They should understand CS/CE core fundamentals but they don't need to know how to admin.
Oh wow, so uh... I'm managing around 1000 nodes over 6 clusters, alone. There's others able to handle things when I'm not around or on leave and meticulously updated docs for them to do so but in general am the only one touching our infra.
I also do dev work the other half of the week for our company.
Ask your boss if he needs a hand :)
How many different applications/services are you running?
In any case, absolutely amazing what one person can manage with modern infrastructure.
At one job I was the only IT person and we had ~250 plain boring VMs on some bare metal Linux/KVM hosts. No config management. No Kubernetes. I fixed that quickly. There was one other guy capable of taking a look at most of it.
I was also doing the software builds and client releases, client support, writing the documentation for the software, and fixing that software.
I suspect we would have had no problem scaling up with some better tooling. Imagine a team of 150? When people tell me things like that, it sounds more like the solution isn't much of a solution at all.
Hehe, you lack skill in empire building. You know "leading a team of highly motivated team of 50+ devops engineers". The kind of talent that postpones patching until you are back from vacation. Or deploying config change that needs at least two rollbacks before finally going in.
Because 100-150 for the devops would be crazy for a mid-sized system like that.
Unless you're managing Windows servers or something.
Have you been in a company with ~2000+ servers where devs install their apps on these OSs and building packages that refuse to upgrade to the latest OS? I mean even with LTS a 20 year old company may still have 3-4 LTS OSs because that last 5% refuse to or cannot upgrade their application to work with the new OS. Sure you could VM the entire thing, but Docker + K8s removes that completely.
Also, intricate linkage between an app and the host OS also means there’s more work involved with upgrading.
THats what ld_preload is for. But real talk, if you have 2k servers and you can't package your apps to run on your OS, then you need a different platform team.
We managed 36k servers using fucking salt and perl. We were packaging nvidia drivers and all sorts. One system that everyone used still needed the athena widget set.
But the main point is, if you're using old packages, then you're gonna get hacked. You either need to kill that app, fire that developer or virtualise it and fill out the risk register, and do monthly recovery tests.
Docker allows you to pack in CVEs like no tomorrow. so sure k8s can let you do that, and given the hardly anyone properly enforces namespace isolation, so they can have a service mesh, you can still steal loads of data from a compromised container.
Kubernetes is for rather special case environments. I am coming around to the idea of using Kubernetes more, but I still think that if you're not provisioning bare-metal worker nodes, then don't bother with Kubernetes.
The problem is that Kubernetes provides orchestration which is missing, or at least limited, in the VM and bare-metal world, so I can understand reaching for Kubernetes, because it is providing a relatively uniform interface for your infrastructure. It just comes at the cost of additional complexity.
Generally speaking I think people need to be more comfortable with build packages for their operating system of choice and install applications that way. Then it's mostly configuration that needs to be pushed and that simplifies things somewhat.