"Premature optimization is the root of all evil" -- Donald Knuth
Translating the old suggestions [1] to the realm of devops, I think the point really is: if you are fairly certain that your optimization (k8, docker and so on) will result in "better" code and practices right away, then you should do it. If not, you shouldn't.
I personally find this stuff to be way overkill in a lot of cases. Does Kubernetes really accelerate your development process? If you are a two-men startup, your objective is to find ways to deliver value as quickly as possible, not to play at being Facebook. When you become Facebook (or even just Basecamp), you will then have enough resources to do this optimization. But if you feel your development process is really so much better with K8 and friends (because it's what you used in a previous job or something), by all means go for it.
The number of projects that might scale up is much, much smaller than the number of projects that do. If I actually want my project to serve a zillion users, the right place for me to focus my effort is not on Kubernetes, but on user context interviews, user tests, and fast iteration based on the results of experiments.
I tried it about 6 months back with the intent of using it in a corporate prod environment and getting set up was... a massive pain in the ass to say the least - compared to the existing ansible set up. It was supposed to solve headaches, not cause them.
I wasn't impressed. I wouldn't be surprised if it ends up being "Angular 1.0" to someone else's react.
last year took, i think 2 days. my private one was up and running within ~1h, including writing the ansible role to first install binaries/dependencies and join the cluster as a worker node.
either you didn't use kubeadm to set it up or ... i have no idea how you could've possibly failed.
its pretty much
(all) ${packagemanager} install docker-ce kubectl kubelet kubeadm
(master) kubeadm init -> prints token
(node) kubeadm join ${token}https://github.com/kubernetes/kops
Be aware of this bug at scale ... its interesting
https://blog.quentin-machu.fr/2018/06/24/5-15s-dns-lookups-o...
I did use kubeadm. It required considerably more than just 3 simple steps required to get a basic working cluster up. Two days was more like it.
Personally I really like Docker Compose files - virtually no overhead to maintain alongside a handful of Dockerfiles coupled with really simple syntax for expressing the relationships between them. Containers themselves seems conceptually challenging for some newcomers in my experience (concepts like image immutability etc don't have many similar analogues I'd argue if you are new to containers, and I've seen some very experienced developers get stuck trying to map them conceptually to VM images which is not a good fit), but the payoff in the ease of deploying your work is huge. It's nothing a reasonable developer can't learn in a few hours though, and the documentation is pretty good.
I especially find Docker great for projects I worked on years ago - I no longer really need to keep track of how to install/configure the side project I haven't touched in age's software dependencies, just hit 'docker-compose up' and I'm running.
I also really like Docker-Compose as a replacement for things like Vagrant scripts to create developer environments in some scenarios - way less overhead than a vagrant script spinning up multiple VMs on your laptop, and generally much faster deployments.
Kubernetes doesnt manage the machines. It manages the applications on machines that are managed with something else.
You have to do something else to manage the machines
The question isn't really whether you need dozens of machines, it's whether you can foresee eventually maybe needing dozens of machines.
Remember the bad old days when people said that relational databases were worthless because they "don't scale", that using Mongo and other NoSQL databases were practically a necessity for doing anything modern and "web-scale" because otherwise after you got your big break and you got popular you would need to keep up with all the new traffic and not crash? A lot of engineers have this tendency to worry about scalability long before it's ever a problem. Something about the delusions of grandeur incurred by people who got into engineering because they were inspired by great people building big things.
Starting out by running Kubernetes on a three-node cluster is actually the correct call for a small project if you can reasonably foresee needing to elastically scale your cluster in the future, and don't want to waste days or weeks porting to Kubernetes down the line to deal with your scalability problems that you foresaw having in the first place.
Again, that doesn't mean that Kubernetes is right for every hobbyist project. But there is definitely a (small) subset of hobbyist projects for which it is not overkill.