In regards to something like Kubernetes, i largely agree.
However, there is (currently) certainly some merit to just using Docker, Docker Compose or even Docker Swarm, all of which are relatively boring and stable.
Those essentially solved the problems of environments, configuration, as well as backups for me. Most *nix distros have humiliatingly failed at having a single directory for everything an application needs to run, which horribly complicates figuring out where your data actually lives and how you could re-create everything from 0 in case of a failure or migration. The same goes for different system packages and even sometimes pinning versions. The same goes for networking being weird and is especially applicable in situations where you would want different versions of the same package running on the same server (e.g. MySQL 5.7 for one project, MySQL 8 for another).
A lot of that was already solved by VMs and tools like Vagrant, but it was a bit heavy and never caught on - Docker essentially solved that problem by allowing you to reason about your applications like applications on a phone - just a package of stuff that you care about, adding some resiliency and load balancing capabilities on top.
And then things like Kubernetes came along and addressed more enterprise concerns that are applicable at scale and inadvertently pulled along a bunch of people for the ride, when they have neither the need nor the capacity to work on such complicated software.
I actually talked more about my experiences with this in my blog post, "My journey from ad hoc chaos to order (a tale of legacy code, services and containers)": https://blog.kronis.dev/articles/my-journey-from-ad-hoc-chao...
That said, it's nice if you don't need to solve the problems that containers do. Alternatively, if you have other ways of solving them (e.g. the ansible + systemd setup), then go ahead and use that until your needs expand!
The Filesystem Hierarchy Standard actively militates against a single-directory setup. Instead it is designed so you can keep your static stuff on one read-only location, your interim application runtime stuff on another (/var), your user data on a third (/home)...
It's a high-end standard appropriate for serious computers where these directories can be on different disks, with different filesystems, security protections, performance implications, and backup strategies (no need to back up what you can simply reinstall!). As a small-time user it's only your choice because it's the standard choice.
I guess you can have /opt/xyz?
Thanks to Docker i can mostly reject its conventions (for actually organizing my application data).
Instead, i might have something like the following file paths and container bind mounts:
PATH ON FS --> PATH IN CONTAINER
/docker/app_foo/data/container_x/var/www/html --> /var/www/html
/docker/app_foo/data/container_y/var/lib/postgresql/data --> /var/lib/postgresql/data
/docker/app_bar/data/container_w/app/data --> /app/data
/docker/app_bar/data/container_z/var/lib/mysql --> /var/lib/mysql
*the docker/app_X directories can contain deployment descriptors, instructions or other notes if need be
The hosts can have a non-standard folder at FS root under which all of the persistent data will live, thus backups become easy, and due to organizing stacks/apps in a tree structure, everything that's needed by a particular environment is also immediately obvious.
Furthermore, this doesn't get in the way of the actual OS operating, nor does the OS get in the way of my deployments - if anything ever goes wrong, i can just wipe and reinstall, setup with Ansible and carry over the /docker directory as necessary. Of course, depending on how everything is mounted, it might as well be just reconnecting another drive or a few drives and setting up fstab.
I think that treating your entire system as one large entity is a horrible idea - instead you should have a clear distinction between what's a part of "infrastructure software" and what's your "business software", those two should be kept as separate as possible. The former should be easy to wipe and update and even swap out altogether (within reason), the latter should be treated as critical and backed up thoroughly and often.
Edit: my point is that most existing software is really bad at this distinction and there are far too many opinions and as a consequence your Tomcat install is strewn across a dozen folders, same with many other pieces of software. So why should you live in fear of forgetting some important configuration file or some data directory?
Of course, YMMV, but things like Ansible and Docker (or Podman, or containerd) with the "infrastructure as code" and "cattle instead of pets" approaches is pretty serviceable.
Switch to K3S, you won't regret it. I was hesitant to pour more work into my project to switch to K8S (which I know well, it's part of my day job !)... K3S is an excellent way to get into Kubernetes. Or use Digital Ocean's or other hosted implementations...
Kubernetes has a lot to offer... peace of mind is one of them !
K3s is pretty great, at least as long as you keep on the happy path.
In practice, that probably means a DEB based distro, quite possibly Ubuntu LTS and something like Rancher for management (if you prefer a UI of some sort and can afford the spare RAM), or Portainer, or even just Lens.
Sadly, i had a situation where i needed Kubernetes on a server at work, which had about 8 GB of RAM, so OpenShift was out of the question and most other "full" K8s distros also weren't viable, when that same server was also to run the actual containers. Moreso, i was stuck with Oracle Linux, on which getting K3s working properly was a bit problematic (though the same probably applies to most RPM distros).
Not only that, but instead of Let's Encrypt, i needed to use custom SSL/TLS certificates, installing which with the Traefik ingress and using them as the defaults instead of the self-signed ones was quite the mess, about which i wrote in another comment here on HN: https://news.ycombinator.com/item?id=30672765
In short, to get it working i needed:
- a ConfigMap for Traefik, knowledge about the structure of the ConfigMap (tls.stores.default.defaultCertificate)
- a TLSSecret for storing the actual certificate/key
- a TLSStore (which i also needed to actually use the secret, spec.defaultCertificate.secretName)
- a HelmChartConfig for Traefik to load the ConfigMap with the mounted secrets and config
Oh, attempting to use Rancher also failed due to recent changes with cgroups v2 and varying support for all of that, whereas it seemed to work properly on a throwaway Ubuntu LTS VM/VPS.
In short:
- i still think that the goals of the K3s project are really nice, the current resource usage is surprisingly decent
- if you have to use Kubernetes on RPM, go with OpenShift, if you don't have the resources for it, just use DEB distros
- the struggle with niche setups, like i needed, is largely not worth your time, the documentation for these still isn't there yet (example of mature documentation: Apache, which has been around for decades)
- ideally, just pay someone else to run a cluster for you, if you can (i can't, because mostly on-prem at work and am relatively poor in regards to personal homelab)
But I must admit it's easy/easier when someone else is managing K8S for you, be it DigitalOcean or Tencent (which I have experience with) or AWS / Google Cloud.
K3S is quite comparable to Docker Swarm though. I liked Swarm when I developed SSLPing because of its simplicity compared to K8S, but once you learn your way through K8S, there's no point sticking to Swarm I think...
If you wanted an 'in' to Kubernetes and better automation of your homelab. I would suggest to check out this repo[1], it's nearly a A-Z guide on getting k3s running on Ubuntu (I am no distro-snob, Ubuntu just works well for most home workloads in Kubernetes). GitOps tools like Flux and Argo are really becoming popular to use with Kubernetes because all your configuration is stored in Git and the GitOps tools deploy manifests based on the Git repo state. My entire home cluster[2] is open source and there's many who are also doing something similar[3].
All of this comes at a cost of learning these tools which is not easy, but from my interactions with people who do take the plunge from docker-compose or docker-swarm most of them stay and once they see the benefits of GitOps and Kubernetes cannot go back to their previous ways. Automation is king and it is much easier using Kubernetes because most of the tools exist for it already.
1. https://github.com/k8s-at-home/template-cluster-k3s
As someone who is still running all of their homelab stuff (well, almost) on Docker Swarm, i dread the day when the eventual switch to Kubernetes will be inevitable.
Maybe Nomad will save me.
Or something like K3s or even Rancher - even though the former has some ingress weirdness with Traefik and default wildcard SSL/TLS certificates since not everyone uses Let's Encrypt or wants DNS challenge for private stuff and the latter needs a lot of resources to even run.
Still, when things go wrong with Docker Swarm, they're generally pretty easy to debug and solve - it's like adding just a little bit more on top of Docker Compose. Worst issues i've had were related to networking and kernel updates breaking something, though purging the node and carrying over all of the bind mount data as well as re-deploying the stacks solved that.