I work for Red Hat as an OpenShift Consultant, so I'm on the bleeding edge of change and am constantly pushing boundaries.

Don't tell the boss or the customers, but most of the time when release notes for a new version come out, I look at them and go "WTF, why do we need that, I better do some research." It's fast changing, complex as hell, and absolutely brutal. That said most things are there for a reason and once I dig in I usually see the need.

That said I do love it despite its warts. There's no doubt some Stockholm Syndrome at play here, but I love the API (which is pretty curlable btw, a mark of a great API IMHO) and the principles (declarative, everything's a YAML/JSON object in etcd, etc). I see it the same way I did C++ (which I also loved). It gives you great power which you can use to build an elegant, robust system, or you can create an unmaintainable, complex, monster of a nightmare. It's up to you.

The full-time job of keeping up with Kubernetes was published in February 2018[0] and things have only gotten faster since then.

[0] https://goteleport.com/blog/kubernetes-release-cycle/ , HN discussion: https://news.ycombinator.com/item?id=16285192

Getting Kubernetes itself running on bare metal ("running" as in "you have containers and can access them") is half a day of work.

What is deadly difficult is getting networking to work. Even a comparably "easy" thing with a couple of two-NIC machines (one external, internet-routable, one DMZ) cost me a fucking week.

What's even worse is when one has to obey corporate restrictions - for example, only having external interfaces on "loadbalancer" nodes:

- First of all, MetalLB only has one active Speaker node which means your bandwidth is limited to that node's uplink and you're wasting the resources of the other loadbalancers.

- Second, you can taint your nodes to only schedule the MetalLB speaker on your "loadbalancer" nodes via tolerations... but how the f..k do you convince MetalLB to change the speaker node once you apply that change?!

- Third, what do you do when you want to expose N services but only have one or two external IPs? DC/OS was way more flexible, you had one set of loadbalancers (haproxy) that did all the routing, and could run an entire cluster on four machines - two LBs, one master, one worker. There is no way to replicate this with Kubernetes. None.

Yes, it's the networking that never works for me either. I'm one guy, wear many hats, and don't have time to chase rabbits down holes. If I follow published instructions and it doesn't work, I pretty much stop there.

Give nomad, docker swarm, or lxd a shot.

I'm currently involved in an effort to rip out docker swarm at work because its overlay networks are shockingly unreliable. LXD looks interesting but https://linderud.dev/blog/packaging-lxd-for-arch-linux/ convinced me that it's another Canonical NIH special and probably best avoided (in particular, "only distributed for Ubuntu via snaps" means "forces auto-updates" which means "not going in my environment"). Need to try Nomad; I'm cautiously optimistic since the rest of HashiCorp's stuff is good.

Nomad 1.0 is the bomb. I've ran Nomad since 0.7 and it was usable but pretty rough in those days. 1.0 is such a good, smooth thing. Super awesome even if you don't want to use Docker at all, you can just execute random stuff through exec/raw_exec, or straight Java. Heck, run it in a BSD homelab and use one of the jailshell executors.

It's not Kubernetes but it doesn't try to be at all.

Is this on 19.x swarm mode? I thought it got a lot more reliable with that release.