Out of all of those, my experiences have only been positive with K3s (though i haven't tried k0s) and RKE, especially when using the latter with Rancher ( https://rancher.com/ ) or when using the former with Portainer ( https://www.portainer.io/solutions ).
In most of the other cases, i've run into a variety of problems:
- different networking implementations that work inconsistently ( https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy )
- general problems with networking, communication between nodes, and certain node components refusing to talk to others and thus nodes not allowing anything to be scheduled on them
- large CPU/RAM resource usage for even small clusters, which is unfortunate, given that i'm too poor to afford lots of either
- no overprovisioning support (when i last tried), leading to pods not being scheduled on nodes when there's plenty of resources available, due to unnecessary reservations
- problems with PVCs and storing data in host directories, like how Docker has bind mounts with few to no issues
- also, they recently stopped supporting the Docker shim and now you have to use containerd, which is unfortunate from a debugging perspective (since a lot of people out there actually like Docker CLI)
If i could afford proper server hardware instead of 200GEs with value RAM then i bet the situation would be different, but in my experience this is a representation of how things would run in typical small VPSes (should you self host the control plane), which doesn't inspire confidence.
That said, when i compared the RAM usage of Docker Swarm with that of K3s, the picture was far better! My current approach is to introduce clients and companies to containers in the following fashion, if they don't use the cloud vendors: Docker --> Docker Compose --> Docker Swarm --> (check if there's need for more functionality) --> K3s.
The beautiful think is that because of the OCI standard you can choose where and how to run your app, as well as use tools like Kompose to make migration very easy: https://kompose.io/
Or perhaps one of the lightweight ones, like K3s ( https://k3s.io/ ) or k0s ( https://k0sproject.io/ )?
Maybe a turnkey one, like RKE ( https://rancher.com/products/rke/ )?
Or did you use something like kubespray ( https://kubespray.io/#/ ) to create a cluster in a semi automated fashion?
Alternatively, perhaps you built your cluster from scratch or maybe used kubeadm ( https://kubernetes.io/docs/setup/production-environment/tool... ) and simply got lucky?
Out of all of those, my experiences have only been positive with K3s (though i haven't tried k0s) and RKE, especially when using the latter with Rancher ( https://rancher.com/ ) or when using the former with Portainer ( https://www.portainer.io/solutions ).
In most of the other cases, i've run into a variety of problems:
If i could afford proper server hardware instead of 200GEs with value RAM then i bet the situation would be different, but in my experience this is a representation of how things would run in typical small VPSes (should you self host the control plane), which doesn't inspire confidence.That said, when i compared the RAM usage of Docker Swarm with that of K3s, the picture was far better! My current approach is to introduce clients and companies to containers in the following fashion, if they don't use the cloud vendors: Docker --> Docker Compose --> Docker Swarm --> (check if there's need for more functionality) --> K3s.
The beautiful think is that because of the OCI standard you can choose where and how to run your app, as well as use tools like Kompose to make migration very easy: https://kompose.io/