That's the same with.. the Linux kernel? I'd wager that most services people write have less complexity than the the kernel it runs on. We choose to do that not because we need its full feature set, but because it gives us useful abstractions.

I don’t know that that’s the right analogy. End users don’t have to write a mountain of configuration in order to effectively use the Linux kernel.

If k8s were as complex as it was, but didn’t force that complexity onto the user, I’d agree.

You can deploy quite a bit to k8s with just a few lines of helm installs, and it will work with very little configuration.

Exactly, I think I could have stated it better. Using k8s is strictly more complex from the perspective of the operator, and the mountain I'm referring to isn't actually from k8s itself but from all the 3rd party stuff you have to layer on to actually make it useful-- things like cluster-autoscaler, vpa, kube-prometheus, aws-load-balancer-controller, misc. operators, that fall on my side of the wall when it comes to fixing it when it's broken.

I can understand that it doesn't come off that way but I actually like k8s and the surrounding ecosystem. But deploying k8s makes your infrastructure as complicated as k8s itself, which can be a massive win because it works in both directions.

If it can’t be monolithic like *BSD, at least offer distributions like the Linux Sphere. Like a ready to go LAMP stack. Or a collection of services (Database, Storage, Logs,…)

There are distributions of k8s for years now. Starting from simple side like k3s, through more complex (k0s, rancher, etc), to complete OS images (Talos, Metropolis)

I’m running Talos locally and run into basically none of these issues — use one of the provided storage providers, dump a container into it, bing bang boom you’re done.

And similarly, you don't need any configuration at all to just run k8s itself, you do need to configure your application and pay attention when exposing things, but the same holds true for an OS on a single machine.

I believe the OS comparison is the most apt, both in the sense of the layer it represents, i.e. abstracting away physical machines and managing resources for applications, but also from what it means: one used to have to rewrite a program for each new computer as they were all unique (incl. CPU), just like each corp had its own way of managing infrastructure, clusters of nodes and automation.

Any new application needed to be integrated from the ground up, which might be simpler in that you only touch what you actually need, but you also have to do it every single time, and it's different for each project/corp - with the vast majority missing out on the more complex bits that don't fit into an nginx config and some shell scripts.

There’s always a mountain of code on top of the cloud provider.

Furthermore, there's always a mountain of code aggregating heterogeneous systems into a uniform interface. Look at the linux kernel, or LLVM, or pretty much any other (successful) project that provides abstractions to many different backends.

Yes but that code isn't typically running in production the way k8s is. Having a lot of terraform is just a fancy method of statically provisioning infrastructure. The code is gone once it applies. Bringing k8s into the mix is a whole other application layer that's providing services to your code. Things like sgs for pods, cni plugins, coredns, is code that wouldn't be there in a non k8s world.

Your problems now include k8s networking as well as VPC networking.

> The code is gone once it applies

Until you need to update a change or upgrade something or patch a security issue. The most dangerous code is code that doesn't run often. Like backup restores. Ticking time bombs unless you're really careful.

> Bringing k8s into the mix is a whole other application layer that's providing services to your code. Things like sgs for pods, cni plugins, coredns, is code that wouldn't be there in a non k8s world.

The same applies to everything running around your tech stack that you didn't write. That includes the language runtime, the compiler, the operating system, all OS libraries you use, and the libraries your app imports. The ship has sailed and K8S isn't much of an addition.

I haven't hit a CNI bug in probably 5 years and that was only on Amazon's horrible CNI code. I've hit a ton of bugs in my software stack. And the libraries I depend on. And my deployment stack. And my cross-service networking stack at the application layer.