AnxiousCoward

I wouldn't call it a misperception. It does orchestration, service discovery, resilience, secrets management (which is a joke, but still) and whatnot. That's definitely a lot of complexity. Mixing it all together doesn't help with making it easier to manage, on the contrary.

Once you get to a place where you need to run really-really huge workloads, like thousands or tens of thousands of machines, kubernetes, hosted or not, will take the will to live away from you. It's not about the learning curve, or the convenience, it's that kubernetes gets slow to catch up on large workloads that need to change often. That kind of setup is where nomad + consul + vault + terraform shine.

I wouldn't consider any of these for production.

May be. But operators are a relative pain to develop. I prefer the HCL approach. You can have modules instead of operators, and while HCL overall might be painful, it's less painful, IMO, than constantly changing operators. My team spends a significant amount of time chasing deprecations of operators and associated helm charts.

You may not use it for automation. But you usually do much exploratory work before doing the automation. That's where the various CLIs for all kinds of things come in really handy.

We had to add vault to our setup even if we use kubernetes - kubernetes secrets are a joke. We don't use consul, but managing istio, or any other service mesh, on top of kubernetes, isn't any simpler than running your own consul. (In fact, I have found istio to be a lot more difficult to get to do what you need than consul's service mesh, and with less functionality.)

You got that sub-1000 boxen wrong. Nomad is usually employed where the infrastructure is huge. IMO, K8s is driven by hype and marketing, not by technical merit.

Your enumeration of the managed things is very particular, in that it includes only stores. There's a reason for things that have keeping persistent state as their reason to exist are managed: reliably maintaining persistent state in the cloud is a lot more difficult than reliably orchestrating things that just have to run.

K8s is complicated because it does many things, not because each of the things it does, individually, is complicated. If your orchestrator only runs stuff, if your service discovery is independent of the orchestrator, and just does the wiring and routing, if your secrets management is simply a thing that provides secrets to authorized clients, a lot of the complexity associated with kubernetes goes away. The whole is always more than the sum of its parts, and in the case of kubernetes this adds up in an unfavorable way, complexity-wise.

IME, a lot of complexity, in kubernetes land, also comes from kubernetes being highly opinionated. Whenever your needs don't match kubernetes' opinions, you have to fight it. To give you an example, service versioning is not something kubernetes knows about. If you need it, you need to add it manually. You can easily automate this with consul, and make applications unaware of service versions. Reciprocal trust verification between ... things that communicate is another. Kubernetes provides nothing. If someone manages to inject a pod into your namespace, and randomly starts replaying requests, there's no automatic way to verify the caller's identity - you have to do this manually, at the application level, using a custom sidecar (if you're lucky, your service mesh has something to help you). If you use consul with nomad, you get these things out of the box.

I believe your life is simple now because what you manage isn't overly complicated. Once you have thousands of distinct deployments, versioned, across several clusters, and thousands or even tens of thousands of pods running in tens of data centers, life with kubernetes becomes a lot more difficult. Of course, you can still manage, by automating everything, but automation, at that level, with kubernetes, is not an easy thing. Or at least that's my experience.

In reality, this is always an illusion, IME. You always have to deal with the platform yourself. You can hire a team to do this dealing with you, but that's not cost effective.

Having a rollback feature at the orchestrator level is useful when you want the rollback to be fast. Sometimes that's critical - when you deploy something buggy in production.

One reason I'd prefer to use Nomad over K8s is scalability. K8s becomes slow and occasionally acts weird with just a few thousand pods. (Not that it is particularly fast with small numbers of pods.) Nomad is known to run reliably in production with many thousands of mixed workloads, not just containers.

Another reason is its specialization. I'd rather deal with a handful of independent, well documented components (consul, vault, nomad, basically) than with one that does all things, but in a way I have to occasionally fight, and which, by necessity, given its breadth, is awfully documented. We do use k8s but still run vault for secrets - k8s secrets are a joke. We don't use consul because our discovery needs are simple, but the service abstraction in k8s is weak at best.

While I'm not a big fan of HCL, any version, either, I do think that being able to manage everything, from cluster configuration to application deployment, via the same tool (terraform), in git, is more convenient than still having to use terraform for infrastructure but being forced to use helm on top of it - you might be able to maintain applications as terraform HCL scripts, but it would be unreasonable, given that for many applications that you'll want to deploy there are readymade charts available.

From a strictly theoretical point of view, designing any kind of software the way kubernetes is designed is bad practice. It consists of very few components - but which do many different things. Some people work hard to maintain the system usable, but IMO the bad structure shows in how users need to interact with the system.

Which is why, if the choice was mine, and not hype-driven, I'd take managed nomad with consul, vault, terraform and some git repo (my choice: gitlab, because it comes with a nice integrated CI) already integrated any time over managed kubernetes. But that's just me.