Kubernetes is for rather special case environments. I am coming around to the idea of using Kubernetes more, but I still think that if you're not provisioning bare-metal worker nodes, then don't bother with Kubernetes.
The problem is that Kubernetes provides orchestration which is missing, or at least limited, in the VM and bare-metal world, so I can understand reaching for Kubernetes, because it is providing a relatively uniform interface for your infrastructure. It just comes at the cost of additional complexity.
Generally speaking I think people need to be more comfortable with build packages for their operating system of choice and install applications that way. Then it's mostly configuration that needs to be pushed and that simplifies things somewhat.
> Generally speaking I think people need to be more comfortable with build packages for their operating system of choice and install applications that way. The it's mostly configuration that needs
why, it’s 2025, docker / container makes life so easy
They should understand CS/CE core fundamentals but they don't need to know how to admin.
Oh wow, so uh... I'm managing around 1000 nodes over 6 clusters, alone. There's others able to handle things when I'm not around or on leave and meticulously updated docs for them to do so but in general am the only one touching our infra.
I also do dev work the other half of the week for our company.
Ask your boss if he needs a hand :)
How many different applications/services are you running?
In any case, absolutely amazing what one person can manage with modern infrastructure.
At one job I was the only IT person and we had ~250 plain boring VMs on some bare metal Linux/KVM hosts. No config management. No Kubernetes. I fixed that quickly. There was one other guy capable of taking a look at most of it.
I was also doing the software builds and client releases, client support, writing the documentation for the software, and fixing that software.
I suspect we would have had no problem scaling up with some better tooling. Imagine a team of 150? When people tell me things like that, it sounds more like the solution isn't much of a solution at all.
Hehe, you lack skill in empire building. You know "leading a team of highly motivated team of 50+ devops engineers". The kind of talent that postpones patching until you are back from vacation. Or deploying config change that needs at least two rollbacks before finally going in.
Because 100-150 for the devops would be crazy for a mid-sized system like that.
Unless you're managing Windows servers or something.
Have you been in a company with ~2000+ servers where devs install their apps on these OSs and building packages that refuse to upgrade to the latest OS? I mean even with LTS a 20 year old company may still have 3-4 LTS OSs because that last 5% refuse to or cannot upgrade their application to work with the new OS. Sure you could VM the entire thing, but Docker + K8s removes that completely.
Also, intricate linkage between an app and the host OS also means there’s more work involved with upgrading.
THats what ld_preload is for. But real talk, if you have 2k servers and you can't package your apps to run on your OS, then you need a different platform team.
We managed 36k servers using fucking salt and perl. We were packaging nvidia drivers and all sorts. One system that everyone used still needed the athena widget set.
But the main point is, if you're using old packages, then you're gonna get hacked. You either need to kill that app, fire that developer or virtualise it and fill out the risk register, and do monthly recovery tests.
Docker allows you to pack in CVEs like no tomorrow. so sure k8s can let you do that, and given the hardly anyone properly enforces namespace isolation, so they can have a service mesh, you can still steal loads of data from a compromised container.
Which solutions do you have in mind?
- VPS with software installed on the host
- VPS(s) with Docker (or similar) running containers built on-host
- Server(s) with Docker Swarm running containers in a registry
- Something Kubernetes like k3s?
In a way there's two problems to solve for small organisations (often 1 server per app, but up to say 3): the server, monitoring it and keeping it up to date, and the app(s) running on each server and deploying and updating them. The app side has more solutions, so I'd rather focus on the server side here.
Like the sibling commenter I strongly dislike the configuration management landscape (with particular dislike of Ansible and maintaining it - my takeaway is never use 3rd party playbooks, always write your own). As often for me these servers are set up, run for a bit and then a new one is set up and the app redeployed to that (easier than an OS upgrade in production) I've gone back to a bash provisioning script, slightly templated config files and copying them into place. It sucks, but not as much as debugging Ansible has.
yes the possix shell is not a good language which is why thinks like perl, python and even php or C got widely used but there is a intermediate layer with tools like fabric(https://www.fabfile.org/) solving a lot of the problems with the fully homegrown without locking you into the "Infrastructure as(manually edited) Data" paradigm that only really works for problems of big scale and low complexity which is exactly the opposite of what you see in many enterprise environments.
E.g., Chef Software, especially after its acquisition, is just a dumpster fire of weird anti-patterns and seemingly incomplete, buggy implementations.
Ansible is more of the gold standard but I actually moved to Chef to gain a little more capability. But now I hate both of them.
When I just threw this all in the trash in my HomeLab and went to containerization it was a major breath of fresh air and resulted in getting a lot of time back.
For organizations, of the best parts about Kubernetes is that it’s so agnostic so that you can drop in replacements with a level of ease that is just about unheard of in the Ops world.
If you are a small shop you can just start with something simpler and more manageable like k3s or Talos Linux and basically get all the benefits without the full blown k8s management burden.
Would it be simpler to use plain Docker, Docker Swarm, Portainer, something like that? Yeah, but the amount of effort saved versus your ability to adapt in the future seems to favor just choosing Kubernetes as a default option.
If you are in the position to pick a config management system, the best you can do is to chart out your current and known upcoming use cases. Then choose the tool that sucks the least for your particular needs.
And three years down the line, pray that you made the right choice.
Yes, kube is hideously complex. Yes, it comes with enormous selection of footguns. But what it does do well, is to allow decoupling host behaviour from service/container behaviour more than 98% of the time. Combined with immutable infrastructure, it is possible to isolate host configuration management to the image pre-bake stage. Leave just the absolute minimum of post-launch config to the boot/provisioning logic, and you have at least a hope of running something solid.
Distributed systems are inherently complex. And the fundamental truth is that inherent complexity can never be eliminated, only moved around.
It's also basically a standard API that every cloud provider is forced to implement, meaning it's really easy to onboard new compute from almost anyone. Each K8s cloud provider has its own little quirks, but it's much simpler than the massive sea of difference that each cloud's unique API for VM management was (and the tools to paper over that were generally very leaky abstractions in the pre-K8s world).
So instead of an ansible playbook/role that installs, say, nginx from the distro package repository, and then pushes some specific configuration, I have a dockerfile that does the same thing? Woohoo?
Other issues like secrets and environment management is something I find way more annoying using a tool like Chef.
Try doing a chef policyfile bootstrap that gets some secrets using its own built in chef vault. You can’t do it without wild workarounds because the node isn’t granted access to secrets until it becomes a registered node, and it doesn’t register until a chef client run completes successfully. It’s a really dumb catch-22 design.
The solution is “just use a big 3 cloud secrets vault or Hashicorp vault” and that’s fine but it’s really strange that the tool can’t handle something so simple on its own.
I fully accept that there are sizes and complexities where k8s is a reasonable choice, and sometimes it's a reasonable choice because it's easier to hire for, but the bar should be a lot higher than what it currently is.
It's a reason why I'm putting together alternatives for those of my clients who wants to avoid the complexity.
We have Configuration Management systems like Puppet in mature enough state for over a decade now.
I haven't installed server manually or "with handmade scripts" in good 12 years by now.
We have park of around 100-200 servers and actually managing hardware is tiny part of it
> Now, k8s has engulfed a lot more than the primary target, but smaller shops go for it because they'r also hoping to hit it big someday I guess. Otherwise, there will be far easier solutions at lower scale.
K8S is popular because it gives developers a lot of power to deploy stuff, without caring much at underlying systems, without bothering ops people too much. Cloud-wise there is a bunch of native ways to just run a few containers that don't involve it but onprem it is nice way to get a bit faster iteration cycle on infrastructure, even if complexity cost is high.
It is overkill for I'd imagine most stuff deployed in K8S and half of deployments are probably motivated by resume padding rather than actual need.
Not even that. One repository I checked this week had some commits which messages were like "synchronize code with what is on production server". Awesome. And that's not counting the number of hidden adhoc cronjobs on multiple servers.
Also as a dev I like having a pool of "compute" where I can decide to start a new project whenever instead of having to ask some OPS team for servers, routing, DNS config.
If you were working in the orgs targeted by k8s, I think it was generally more of a mess. Think about managing a park of 100~200 servers with home made bash scripts and crappy monitoring tools and a modicum of dashboards.
Now, k8s has engulfed a lot more than the primary target, but smaller shops go for it because they'r also hoping to hit it big someday I guess. Otherwise, there will be far easier solutions at lower scale.