However, those bugs seem to be fixed after version 19.x, but I'm still bitter about all the problems that I had because of simple features failing on me.
Kubernetes is more of an everything-and-the-kitchen-sink approach that you can’t really outgrow but because of its monolithic design, debugging can be quite challenging. It does, however, come with a much larger and much more active community that you can go to for help.
Swarm is great to get simple orchestration going, but it doesn't really do the same thing.
With k8s you can configure multi-point routing, storage backends and load balancing among other things.
With swarm, you get ... Overlay network (which is shit if you actually need to really scale) and 'scaling', which starts n containers of the same service.
Swarm feels more like an mvp which actually works even for small teams. K8s is more like a behemoth which only dedicated personal can fully tame with a silly amount of features most people don't need.
We've used both at my current job for toy projects (self hosted). Never in production however.
And I'm a personal user of gcp - which works wonderfully .. albeit more expensive than simple root servers
Load balancing should be a solved problem already. Swarm and Kubernetes should be using dead simple off-the-shelf software for ingress and load balancing. Any competitor should be able to use the same solutions. To put it another way, this shouldn't be a differentiator.
The problem is that the functionality in tools like nginx are still tied to static network architectures that evolve slowly, and don't take advantage of things like diurnal variability in workloads.
The nginx ingress controller for example, under the hood, just generates and applies a big ol' nginx config! You can extract it and view it yourself, or extend it with a Lua script if you want to be fancy and do some FFI inline with processing the request, etc.
I learned the hard way that GKE, in using GCP's load balancers don't support the same syntax for Ingress patterns as when you use an nginx Ingress. Definitely read the documentation thoroughly!
Lots of people have different definitions of 'easy' which is why I didn't say 'easy'. But how did you get this far off the rails with 'dead simple'?
Beyond that there's a sea of options for more native integrations that will depend on whether your LB vendor has an K8s integration, how friendly your networking team is, and how much code you're willing to write.
When certificates become a logistics category of their own, you really do need some monitoring software to warn you when certificates are winding down.
Years ago I worked on a complex code signing system and one set of users were adamant we have alerts starting at 90 days out. It took some convincing to get me to agree it was a priority, but some of my coworkers were never convinced.
Now they have a new career.
I would like to know more about what's going on here. Is this just a sloppy description and in fact Kubernetes uses a private PKI, so that the certs you're using aren't in fact self-signed but signed by a private CA?
Kubernetes and cloud native software make a lot of use of TLS for mutual auth.
A standard Kubeadm cluster (very vanilla k8s) has 3 distinct Certificate authorities, all with the CA root keys online.
On top of that things like Helm, FluentD and Istio will make use of their own distinct TLS certs.
One of the most "fun" pieces is that k8s does not support certificate revocation, so if you use client certs for AuthN, then a user leaving/changing job/losing their laptop can lead to a full re-roll of that certificate authority :)
Also recommended: keep api behind a jumpbox.
That said the distributions I've seen that make use of client certs, don't do that (typical lifetime for a client cert is 1 year), so I'm guessing a load of people using k8s will have these certs floating about...
Powerful creds should be limited in lifespan and machine issued to users in a transaction that involves mfa.
We built our own but Gravitational Teleport I think has these patterns in a product.
The other issue even if you get certs right is service tokens (an admin could have grabbed a copy of one and they can be used from outside cluster) so want to keep that api wrapped IMHO.
You might equally say we’re reinventing “authenticated sessions”. More in common with jwt cookies tbh.
We don’t want to run krb infrastructure so we don’t do that.
The runtime dependency on ldap or NIS, plus keeping krb itself HA, fed and happy plus OS dependent PAM setup make krb fairly undesirable in a production cloud environment if ssh certs and kube certs would suffice.
I find that most people confuse or combine "self-signed" with "signed by a private CA". For a lot of uses, the configuration pains are the same to the user: "I have to load this cert into the CA root trust store". They don't realize how much better a private CA really is.
And of course, PKI would be so much more useful with "name constraints" so you don't have to trust a private CA for all domains just the one domain you care about.
Now, whether or not that works for your users...
I would make the taxionomy a bit more precise: failures at running software on the platform and failures at running the platform itself (which of course affect the software running over it: high latency, network packets dropped, DNS not reolved, ...)
In general I find that when the platform works as expected, it is not that easy to make software run on it fail. That is, it is harder than without what Kubernetes provides (granted: you can have it without Kubernetes, but many of us didn't bother to have capable admins setting things up the right way).
What I find extremely fragile is the platform itself. It works, but you have to start and stop things in the right order, navigate a sea of self-signed TLS certificates that can expire, iptable rules and services and logs.
All have failure modes you need to learn: it takes a dedicated team. And once you have that, you'll need another team and cluster to perform an upgrade.
But hey, when it works, it is really cool to deploy software on it.