I'd take a stack of yaml over a stack of bespoke aws managed service scripts any day.
> Yes, the standard approach is more complex. But it is better _in practice_ to have a single approach, rather than many individually-simpler, but in-aggregate-more-complex approaches.
There still remains such a strong reflexive "you don't need Kubernetes for that" response that shows up.
But we really are seeing a lot more people who get the value of there being a standard way of doing things, of having common platform & expectations underneath. The cost of starting up may be significant, but it results in a more legible understandable result, and we can keep reusing that base of capabilities again and again and again.
Its such an exciting change. Open source has been around for a long time, but running software has been the wild West, until very very recently. We now have this awesome, scalable, extensible way to manage containers, volumes, load balancers, databases, and whatever else have we.
Love to see it. Folks aren't wrong about it having complexity, but so much of the complexity already existed, was just not clearly visible or well harnesses (different example is like how systemd makes it so easy to restrict services, give them temporary/unprivileged users). We are closer to a real shared understanding, to having so many particular decisions offloaded & so much more we can work together atop. Love it so much.
If you're already using docker-compose, the jump from there to Docker Swarm is arguably tiny in comparison to the jump from there to Kubernetes, while still bringing most of its value.
So yes, I will keep evaluating the tradeoffs every time and choose accordingly.
There’s another thing that no one who advocates for these systems wants to mention: The cost of maintenance. I’m ok with systemd as 98% is outsourced to the maintainers. But I’d be more comfortable if k8s was a more monolithic system a la BSD. At least linux have distros.
It makes absolutely no sense to base this decision on the number of users. We have some applications that don't even have 10 users but still use k8s.
Try to understand the point that was made in the original comment: Kubernetes is a way to actually make infrastructure simpler to understand for a team which maintains lots of different applications, as it can scale from a deployment with just one pod to hundreds of nodes and a silly microservices architecture.
The point is not that every application might need this scalability, no the point is that for a team needing to maintain lots of different applications, some internal, some for customers, some in private datacenters, some in the cloud, Kubernetes can be the single common denominator.
Hell, I'm a hardcode NixOS fan but for most services, I still prefer to run them in k8s just as it is more portable. Today I might be okay having some service sitting on some box, running via systemd. But tomorrow I might want to run this service highly available on some cluster. Using k8s that is simple, so why not do it from the start by just treating k8s as a slighly more powerful docker-compose.
k3s is essentially a single binary and installed, up and running with a single command.
With a handful of command line flags you can have a full cluster up and running with some master nodes and some agents and high availability through etcd and load balancing and network policies and ingress and distributed clouds via WireGuard/Tailscale… They also have “blessed” options (that you can generally install with a single command) for things like persistent storage, etc.
I would not want to build a k8s cluster from scratch. Setting up and maintaining a k3s cluster is pretty trivial.
Personally, to me, this sounds like madness.
Deciding you need to custom tailor and brew up a specific solution that exactly matches each use case begets endless complexity. Constantly negotiating a position & changing approaches begets endless complexity.
And the complexity needle moves over time, such that your initial assumptions often don't stand. Getting something running usually isn't too hard. But creeping demands of observability, monitoring, rotating certs, backups, and other concerns are going to start getting tacked on. If you only ever do one thing, and there aren't many of ya'll & never will be, and very few newcomers will ever join, maybe maybe maybe you find a win today by doing less, by hand picking your options & getting in there & cobbling your stack of tools together. Convincing yourself to start small, to avoid "complex" Kubernetes to do less, is in most cases though going to backfire even for a single piece of software, as situations evolve.
There is such a fear that Kubernetes is complex, but it just feels like so many are missing the point, are resisting the standards & practices that erulabs' spoke to.
But man, it is so not complex to prop up a big k3s node & push some containers at it. And anyone whose spent more than 1 hours with kube will be able to look around, see what's running on the node, see how everything works; you won't have to explain what you did, it'll be obvious. And they'll be able to keep going. And you'll be able to keep encorporating concerns & platform & new apps in. And those skills - mastering that complexity - will transfer to every other thing you want to do, will scale way up and way down.
(I do think systemd offers a good number of similar advantages of replicatability, consistency, paved-roadness, and is pretty solid an option, but involves more finding-your-practice than kube)
With k8s you can easily install a bunch of helm charts and get a bespoke platform that becomes a full time role for multiple people.
There are pros and cons against this approach but you if you're worried k8s is complex, just use the cloud native integrations.
And not a collection of strung-together controllers.
If k8s were as complex as it was, but didn’t force that complexity onto the user, I’d agree.
I can understand that it doesn't come off that way but I actually like k8s and the surrounding ecosystem. But deploying k8s makes your infrastructure as complicated as k8s itself, which can be a massive win because it works in both directions.
I believe the OS comparison is the most apt, both in the sense of the layer it represents, i.e. abstracting away physical machines and managing resources for applications, but also from what it means: one used to have to rewrite a program for each new computer as they were all unique (incl. CPU), just like each corp had its own way of managing infrastructure, clusters of nodes and automation.
Any new application needed to be integrated from the ground up, which might be simpler in that you only touch what you actually need, but you also have to do it every single time, and it's different for each project/corp - with the vast majority missing out on the more complex bits that don't fit into an nginx config and some shell scripts.
Your problems now include k8s networking as well as VPC networking.
Until you need to update a change or upgrade something or patch a security issue. The most dangerous code is code that doesn't run often. Like backup restores. Ticking time bombs unless you're really careful.
> Bringing k8s into the mix is a whole other application layer that's providing services to your code. Things like sgs for pods, cni plugins, coredns, is code that wouldn't be there in a non k8s world.
The same applies to everything running around your tech stack that you didn't write. That includes the language runtime, the compiler, the operating system, all OS libraries you use, and the libraries your app imports. The ship has sailed and K8S isn't much of an addition.
I haven't hit a CNI bug in probably 5 years and that was only on Amazon's horrible CNI code. I've hit a ton of bugs in my software stack. And the libraries I depend on. And my deployment stack. And my cross-service networking stack at the application layer.
Very much depends on the point of view. It's great from an SRE point of view but not necessarily application/developers, who are being constrained to a platform's idiosyncracies and expressions of platform egos.
The individually simpler solutions are only complex from a high level, ivory tower, or middle management perspective, not from the perspective of people who have to use and manage the application itself.
In the last 6 months my job has been to get production ready vault instances on azure. There's a bunch of complex, unreliable and not very fun APIs here.
Much like AWS, there isn't really a StatefulSet ( PetSet ) abstraction. To you need to write a bunch of bespoke logic where you figure which IP addresses, names, IDs, and Disks you need to attach to a new VM.
Whilst iterating, the azure APIs are eventually consistent that cause all sort of niggly problems. Resources that are created don't get added to TF when there's a failure.
I create a new vault on a PR and it takes 20 minutes to deploy.
The problem took a couple of months and I created a bunch of code.
On k8s I can just deploy the helm chart in 2 minutes. The abstractions are cleaner, it's more reliable, and way more fun.
At the end of the azure project, the team agreed to do a AKS poc, where we gave the task to the junior of the team, and the entire thing was done in a week.
If you're not doing this type of work, maybe you don't need k8s. But if you're not doing ephemeral test environments, do you actually not see them as a positive, or is it an effort thing? Because it takes me no real effort.
Then, it mostly works, so management makes "economy of scales" and removes all screw drivers, since they need more skills anyway.
Now, you are only using nails.
Until someone else rediscovers screws and starts another hype cycle.
"simple/complex" is not the right paradigm. The real SRE controversy is "unique/standard". Yes, the standard approach is more complex. But it is better _in practice_ to have a single approach, rather than many individually-simpler, but in-aggregate-more-complex approaches.
Kubernetes is never the perfect solution to an engineering problem, but it is almost always the most pragmatic solution to a business problem for a business with many such problems.