Such a monoculture has the same issues as MS Windows had, even for the same reasons.
The "Unix Way" of simple tools interacting seems more reasonable. Especially when it comes to lock-in effects.
It's very much built the UNIX way.
Runtimes (CRI) - Docker shim (deprecated), containerd, CRI-O, kata
Networking (CNI) - Flannel, Calico, Cilium, cloud specific ones, many more
Storage (CSI) - way too many device plugins - GPU, TPU, RDMA/SRIOV NICs
Data store - etcd, dqlite in microk8s, SQLite/postgresql/mysql in k3s
DNS - kube-dns (somewhat deprecated), CoreDNS
Ingress - NGINX (multiple), HAProxy, Envoy (many), etc
Kubelet -https://github.com/virtual-kubelet/virtual-kubelet
Kube-proxy - Cilium can act as a replacement
Cloud-controller-manager for each cloud provider
kube-scheduler - https://kubernetes.io/docs/tasks/extend-kubernetes/configure...
kube-apiserver and kube-controller-manager are two parts where I’m not aware of any other implementations but a) they are kind of the heart of k8s and b) can be easily extended with CRDs/operators.
Imagine if you wanted to use a Unix operating system, but instead of choosing a Linux distro, you just read the POSIX standard and went out and found every required utility, plus a kernel, and had to figure out on your own how to get those to work together and create a system that can run application-level software. If you just go to kubernetes.io and follow the instructions on how to get up and running with the reference implementation, that is what you're doing. It makes no decisions at all for you. You can run external etcd, or use kubeadm to set it up for you. You can run it HA or on a single node. You can add whatever overlay network you want. You can use whatever container runtime engine you want. You can use whatever ingress controller you want, or none at all, and not have any external networking, just as you can install Linux From Scratch and not even bother to include networking if you want a disconnected system for some reason.
You have pretty much complete user freedom, and that is, in fact, the source and reason for a whole lot of complaints. Application developers and even most system administrators don't want to have to make that many decisions before they can even get to hello world. I believe Kelsey Hightower commented on this a while back, saying something to the effect that Kubernetes is not meant to be a developer platform. It's a framework for creating platforms.
Application developers, startups, and small business should almost never be using Kubernetes directly unless they're actually developing a platform product. Whether you use a "distro" like RKE2 or k3s or a managed service from a cloud provider, building out your own cluster using the reference kubernetes is the modern day equivalent of deploying a LAMP stack but doing it on top of Linux From Scratch.
- Nomad is a scheduler. Clean and focused. It is very fast. I was an early user and encountered a number of bumps, but that's software. The people at Hashicorp are super sharp and lovely.
- K8s is a lot more. It includes a scheduler, but in the simplest sense, it is a complete control-plane based on the control-loop pattern. You have an API, a scheduler, a db, various controllers, etc. Forget for a moment that most people use it to orchestrate containers -- it's really designed to orchestrate anything. Its API is extensible, you can add and compose controllers -- there are many possibilities once you wrap your head around it.
This is all opinionated and includes a lot of capability. It's just very different.
You can stitch together nomad, consul, vault, and various glue to create a container orchestration system... but when you start wanting to manage the control-planes as though they are the "kind" (the container for example) with meta-control-planes, and you start wanting to orchestrate network, storage, and other dependencies... all while doing this in a multi-tenant environment, then things get interesting.
-charles.