I like to think that most people who are upset at Kubernetes don't hate on all of it. I think the configuration aspect (YAML) and the very high level of abstraction is what get people lost and as a result they get frustrated by it. I've certainly fall in that category while trying to learn how to operate multiple clusters using different topologies and cloud providers.

But from an operational standpoint, when things are working, it usually behaves very well until you hit some rough edge cases (upgrades were much harder to achieve a couple of years back). But rough edges exist everywhere, and when I get to a point where K8s hits a problem, I would think that it would be much worse if I wasn't using it.

> I like to think that most people who are upset at Kubernetes don't hate on all of it. I think the configuration aspect (YAML) …

I question the competence of anyone who does not question (and rag on) the prevalence of templating YAML.

> But rough edges exist everywhere, and when I get to a point where K8s hits a problem, I would think that it would be much worse if I wasn't using it.

Damn straight. It’s only bad because everything else is strictly worse.

Helm isn’t YAML. It’s a go template file that should compile to YAML, masquerading as YAML with that extension.

So yaml formatters break it, humans struggle to generate code with proper indents, and it’s an insane mess. It’s horrendous.

>I think the configuration aspect (YAML)

What are the reasons to not use JSON rather than YAML? From my admittedly-shallow experience with k8s, I have yet to encounter a situation in which I couldn't use JSON. Does this issue only pop up once you start using Helm charts?

One of the most annoying limitations of JSON is that it does not allow for comments.

at the surface level yaml is a lot easier to read and write for a human. less "s. but once you start using it for complex configuration it becomes unwieldy. but at that point json is also not better than yaml.

after using cdk i think that writing typescript to define infra is a significantly better experience

Naw, just use system-d, ha-proxy and bash scripts. That is much "simpler" (for some definition of simple).

Kidding of course. If you need anything approximating Kubernetes, use it. If you just need one machine maybe don't.

This is the gist of it, I believe. Too often I see "sprinkle some k8s" on it to fix a problem

... when the problem was lacking config management. They overshot; use the right tool for the job

The thing is, "kubernetes" doesn't give you that either. You want a LB? Here's a list of them that you can add to a cluster. But actually pick multiple, because the one you picked in AWS doesn't support bare metal.

Bare metal kubernetes is certainly a lot less complete out of the box when it comes to networking and storage but, people can, and often should, use a managed k8s service which provides all those things out of the box. And if you’re on bare metal once the infra team has abstracted away everything into LoadBalancers and StorageClasses it’s basically the same experience for end users of the cluster.

If you're talking about OpenShift on rented commodity compute, maybe. If you're talking about GKE/AKS/EKS or similar, I disagree wholeheartedly; you're then paying several multiples on the compute and a little extra for Kubernetes.

> because the one you picked in AWS doesn't support bare metal

That's just because AWS's Kubernetes offering is laughably bad.

There is huge difference in your experience whether you use Kubernetes via GKE (Autopilot) or any other solution (at least as long you don't have a dedicated infrastructure team).

I dunno, if you have to use GCP to have a good time, then that's a pretty great argument against k8s for a lot of people.

I think "bare" Kubernetes is still a quite nice tool that allows for learning transferable skills across clouds (similar to Terraform). E.g. even if I have to spin up my own nginx-ingress to be able to handle ingress resources, after having learned that initially, I can basically do the same thing across clouds.

It's just that GKE (Autopilot) does a lot of those out of the box for you that, so you get a much easier end-to-end experience for non-admins (= "request resources -> have them instantiated").

When I reflect what Netflix did back in 2010ish on AWS:

* The declarative infra is EC2/ASG configurations plus Jenkins configurations * Client-side load balancing * ASG for autoscaling and recovery * Amazing observability with a home-grown monitoring system by 4 amazing engineers

Most of all, each of the above item was built and run by one or two people, except the observability stack with four. Oh, standing up a new region was truly a non-event. It just happened and as a member of the cloud platform team I couldn't even recall what I did for the project. It's not that Netflix's infra was better or worse than using k8s. I'm just amazed how happy I have been with an infra built more than 10 years ago, and how simple it was for end users. In that regard, I often question myself what I have missed in the whole movement of k8s platform engineering, other than people do need a robust solution to orchestrate containers.

A big chunk was companies that don't have netflix-money having to bin-pack compute for efficiency.

Or at least that's how I got into k8s, because it allowed me to ship for 1/10th the price of my competitor.

There is no easy solution to manage services and infrastructure: people who hate kubernetes complexity often underestimate the efforts of developing on your own all the features that k8s provides.

At the same time, people who suggest everyone to use kubernetes independently on the company maturity often forget how easy it is to run a service on a simple virtual machine.

In the multidimensional space that contains every software project, there is no hyperplane that separates when it’s worth to use kubernetes or not. It depends on the company, the employees, the culture, the business.

Of course there are general best practices, like for example if you’re just getting started with kubernetes, and already in the cloud, using a managed k8s service from your cloud provider could be a good idea. But again, even for this you’re going to find opposing views online.

> There is no easy solution to manage services and infrastructure: people who hate kubernetes complexity often underestimate the efforts of developing on your own all the features that k8s provides.

This. I was trying to create some infrastructure and application once, using various AWS and off the shelf components. I stopped halfway through when I realized I was reinventing k8s, very poorly. That's when I switched gears and learned k8s.

With that said, I use it sparingly due to the inherent complexity it brings, but at least I have a better handle on how and when it should be used and when it should not, and what problems it solved, since I myself was trying to solve some of the problems.

Aren’t you just describing the basic features of an orchestrator?

Docker Swarm has all those features for example.

(Not that I am recommending Docker Swarm.)

> * declarative define your infrastructure

> [...]

> * has a huge amount of pre-packaged software available (helm charts)

> * and most importantly: allows you to stand up mostly the same infrastructure in the cloud, on your own servers (k3s), and locally (KIND), and thus doesn't tie you into a specific cloud provider

NixOS. I have no clue about kubernetes, but I think NixOS even goes much deeper in these points (e.g. kubernetes is at the "application layer" and doesn't concern itself with declaratively managing the OS underneath, if I understand right). The other points seem much more situational, and if needed kubernetes might well be worth it. For something that could be a single server running a handful of services, NixOS is amazing.

I use NixOS, both on servers and on my machines, but it solves a completely orthogonal problem.

Kubernetes manages a cluster, NixOS manages a single machine.

I wouldn't say completely orthogonal. E.g. the points I've cited are overlap between the two, and ultimately both are meant to host some kind of services. But yes NixOS by itself manages a single machine (although combined with terraform it can become very convenient to also manage a fleet of NixOS machines). Kubernetes manages services on a cluster, but given how powerful a single machine can be I do think that many of those clusters could also just be one beefy server (and maybe a second one with some fail over mechanism, if needed).

If the cluster is indeed necessary though, I think NixOS can be a great base to stand up a Kubernetes cluster on top of.

There are lots of native NixOS tools for managing whole clusters (NixOps, Disnix, Colmena, deploy-rs, Morph, krops, Bento, ...). Lots of people deploy whole fleets of NixOS servers or clusters for specific applications without resorting to Kubernetes. (Kube integrations are also popular, though.) Some of those solutions are very old, too.

Disnix has been around for a long time, probably since before you ever heard of NixOS.

Cloud and terraform gives you those.

You’re right that kubernetes is a bit batteries included, and for that its tempting to take it off the shelf because it “does a lot of needed things”, but you don’t need one tool to do all of those things.

It is ok to have domain specific processes or utilities to solve those.

> Cloud and terraform gives you those

* your stack almost always ends up closely tied to one cloud provider. I've done and seen cloud migrations. They are so painful and costly that they often just aren't attempted.

* Cloud services make it much harder to run your stack locally and on CI. There are solutions and workarounds, but they are all painful. And you always end up tied to the behaviour of the particular cloud services

> but you don’t need one tool to do all of those things

To get the same experience, you do. And I don't see why you would want multiple tools.

If anything, Kubernetes isn't nearly integrated and full-featured enough, because it has too many pluggable parts leading to too much choice and interfacing complexity. Like pluggable ingress, pluggable state database, pluggable networking stack, no simple "end to end app" solution ( KNative, etc), ... This overblown flexibility is what leads to most of the pain and perceived complexity, IMO.

> This overblown flexibility is what leads to most of the pain and perceived complexity, IMO.

Huh, I guess you are spot on. My first experience with kubernetes was k3s and I couldn't for a long time figure out what's all the fuss is about and where is all that complexity people talk so much about. But then I tried vanilla kubernetes.

Perhaps a little on the tinfoil hat side of things, but it isn't completely unreasonable to think that some of the FUD could originate from cloud providers. Kubernetes is a commoditizing force to some extent.

Far from it. TF is mostly writing static content, maybe read one or two things. It’s missing the runtime aspect of it, so are most cloud offerings, without excessive configuration. Rollouts, health probes, logs, service discovery. Just to name a few.

You missed what I think is the most important point in OP's list: it does all of the above in a cloud agnostic way. If I want to move clouds with TF I'm rewriting everything to fit into a new cloud's paradigm. With Kubernetes there's a dozen providers built in (storage, loadbalancing, networking, auto scaling, etc.) or easy to pull in (certificates, KMS secrets, DNS); and they make moving clouds (and more importantly) running locally much easier.

Kubernetes is currently the best way to wrap up workloads in a cloud agnostic way. I've written dozens of services for K8s using different deployment mechanisms (Helm, Carvel's kapp, Flux, Kustomize) and I can run them just as easily in my home K8s cluster and in GCP. It's honestly incredible; I don't know of any other cloud tech that lets me do that.

One thing I think a lot of people miss too, is how good the concepts around Operators in Kubernetes are. It's hard to see unless you've written some yourself, but the theory around how operators work is very reminiscent of reactive coding in front end frameworks (or robotics closed loop control, what they were originally inspired by). When written well they're extremely resilient and incredibly powerful, and a lot of that power comes from etcd and the established patterns they're written with.

I think Kubernetes is really painful sometimes, and huge parts of it aren't great due to limitations of the language it's written in; but I also think it's the best thing available that I can run locally and in a cloud with a FOSS license.

Something feels very off and mantra-like with the proportionality of how often cloud migration benefits are being presented as something very important to how often that actually happens in practice. Not to even mention that it also assumes that simpler setups are automatically harder to move around between clouds, or at least that there are a significant difference in required effort.

When I say it's easy to move between clouds, I'm not referring to an org needing to pick up everything and move from AWS to GCP. That is rare, and takes quite a bit of rearchitecting no matter what.

When I say something is easy to move, I mean that when I build on top of it, it's easy for users to run it in their cloud of choice with changes in config. It also means I have flexibility with where I choose to run something after I've developed it. For example I develop most stuff against minikube, then deploy it to GCP or a local production k8s. If I was using Terraform I couldn't do that.

Not sure what kind of apps this is but I can't see the big value-add on a golang app binary wrt to being cloud agnostic, nor wrt local development. It makes even less assumptions on the user's env. Still need some cloud conf (terraform, database etc) either way

If you'll excuse a slight digression, but I think there's a tendency atm to rather pay $1 in extra complexity a 100 times over time, than pay a $5 one-time fee. Like if repeating something similar twice - even if it's easy and not really a lot of effort - is a sign of failure and thus unbearable.

> it does all of the above in a cloud agnostic way.

I'll give you the benefit of the doubt here and say that some of the basics are indeed cloud agnostic.

However, it's plain for many or most to see that outside of extremely "toy" workloads you will be learning a specific "flavour" of Kubernetes. EKS/GKE/AKS etc; They have, at minimum, custom resource definitions to handle a lot of things and at their worst have implementation specific (hidden) details between equivalent things (persistent volume claims on AWS vs GCP for example is quite substantially different).

For multicloud I usually think of my local K8s cluster and GKE, it's been a few years since I touched EKS. I'd love to hear your opinions on the substantive differences you run into. When switching between clouds I'm usually able to get away with only changing annotations on resources, which is easy enough to put in a values.yml file. I can't remember the last time I had to use a cloud specific CRD. What CRD's do you have to reach for commonly?

Thinking about it; the things I see as very cloud agnostic: Horizontal pod autoscaling, Node autoscaling, Layer 4 loadbalancing, Persistent volumes, Volume snapshots, Certificate managment, External DNS, External secrets, Ingress (when run in cluster, not through a cloud service),

That ends up covering a huge swath of my usecases, probably 80-90%. The main pain points I usually run into are: IAM, Trying to use cloud layer 7 ingress (app loadbalancers?)

I totally agree the underlying implementation if resources can be very different, but that's not the fault of Kubernetes; it's an issue with the implementation from the operator of the K8s cluster. All abstractions are going to be leaky at this level. But for PVCs I feel like storageclasses capture that well, and can be used to pick the level of performance you need per cloud; without having to rewrite the common provision of block device.

> and thus doesn't tie you into a specific cloud provider

It ties you to k8s instead, and it ties you to a few company wide heroes, and that is not a 'benefit' as it's being touted here.

Being tied to a cloud is not a horrible situation either. I suspect "being tied to a cloud" is a boogeyman that k8s proponents would like to spread, but just like with k8s, with the right choices, cloud integration is a huge benefit.

Being tied to the cloud is fine if you don’t care about money. Eventually companies do

ever heard of nomad?

“Automatic recovery”

That’s a joke.

Yes... And? We don't have to be happy with our lot if it sucks.

There are an enormous number of tools that meet these requirements, most obviously Nomad. But really any competently-designed system, defined in terms of any cloud-agnostic provisioning system (Chef, Puppet, Salt, Ansible, home-grown scripts) would qualify.

And, for the record, observability is something very much unrelated to kubectl or k9s.

This is how I accomplished these things before. It involved simpler independent pieces which would not collapse whenever something went wrong. They were easy to reason about and building and fixing such tooling did not require me to hire an expensive consultant.

* declarative define your infrastructure

Declare in README.md that we have 3 web servers and that Bob Jones set them up and manages them. Include Bob's email address and phone number.

* gives you load balancing, automatic recovery and scaling

Load balancing via a load balancer or another scheme. DNS is good enough for some cases. There are other solutions.

Automatic recovery - daemon scripts on the box to start all services when the box boots. VPS provider bounces the box when it crashes. That's one. There are others.

Scaling - automatic scaling is not needed at vast majority of companies that are starting out. When we need to scale to 4 servers, change README.md and send Bob Jones a quick message.

* provides great observability into your whole stack (kubectl, k9s, ...)

This is a need that is introduced because of k8s. There's a lot less to observe without k8s, and tools exist for it.

It's like saying "my backhoe has great diagnostic tools for diagnosing backhoe issues." That's true, but I don't have a backhoe and don't need a backhoe for what I am doing.

* has a huge amount of pre-packaged software available (helm charts)

This is a need that is introduced because of k8s. See above.

* allows you to stand up mostly the same infrastructure in the cloud, on your own servers (k3s), and locally (KIND), and thus doesn't tie you into a specific cloud provider

> doesn't tie you into a specific cloud provider

Not a real problem for most companies. If you're preparing to change cloud providers from day one, you are likely spending time on the wrong problem.

> same infrastructure in different envs

This is a benefit, which other solutions come close to, but k8s shines at. You can go a long way without having reproducible multi-machine setups in different envs and can come pretty close when needed, with manual work.

> Kubernetes could have been much simpler, and probably was intentionally built to not be easy to use end to end.

If true, this is a strange design choice. I'd be wary of anything that was made complex just for the sake of it.

k8s gets enough flack without having to accuse it of being complex just for funsies.

> But it's still by far the best we've got.

k8s is the best we've got when we want k8s. The trick is to not want k8s for the sake of wanting k8s.

There are times when k8s provides tremendous value. Most companies who decide to use it do not have the problems that k8s promises to solve, and never will. Sadly, sometimes it's because they've spent their time and money on unnecessary complexity like k8s instead of building a product that delivers value.