E.g. over years of using Etcd. I've come to the conclusion that none of the uses I've actually used it for were necessary, and so I've generally stopped using it. I'll use it again if I come across instances where consistency is sufficiently critical and the system needs to be distributed, but that's a rare situation to be in. E.g. I've seen lots of systems try to use it to distribute configuration data. Most of the time this doesn't need to be consistent, as long as you can determine whether or not it is (and so take outdated instances out of rotation, or not put them back in rotation). Same for load balancer configs, for example.
It's gone to the point where whenever someone mentions "distributed database" alarm bells goes off in my head. 9 out of 10 times it's a sign of someone over-engineering a system and building in unnecessary complexity.
What I want out of a system to manage my containers is something that reduces complexity of deployment and operation, not something that introduces more complexity. So far I've seen nothing of Kubernetes that indicates it fits that bill. Maybe it does on the very high end. My largest systems have "only" been in the few hundred containers/VM range over dozens of servers in 4-5 physical locations (on premises/co-location, managed and public cloud). For that scale, I've personally found Kubernetes overkill and adding too much complexity.
Maybe that'll change some day as it matures, but I'm not in a rush to complicate my stacks.
My main point in all of this is that concepts must be understood to run large distributed applications. If you don't have such an app, nothing applies. If you do, then you have a choice of tools to use, and one of them is Kubernetes which offers a great foundation and set of features. There are other tools like Nomad, Chef or your own code and scripts - it doesn't matter what you use as long as you understand what you're doing.
You have 1000 machines. Each runs 1000 containers with an average runtime of 1 hour per container.
Design a scheduler that runs on a single machine with a cold spare. For failover, assume you have access to a reliable RDBMS or relible NFS server.
Plugging in numbers for my $1000 synology, which runs a half dozen dockers, a vm, and is off-site backed up (for $120/yr), it tells me that I’ll pay $135/month to run comparable kubernetes at google.
Even if you pretend my $75/month broadband connection is only used by the synology, and include power costs, the synology still wins by $10’s/month.
(I included 3TB of storage for kubernetes, and 6TB usable for synology, with 3TB used, and sized for the high-memory 4 core machine, since dram is the synology bottleneck, even with a low wattage cpu. The synology is intentionally over-sized and is basically idle, so “but scalability!” isn’t a valid complaint.)
Also, the 9’s I’ve observed were trouncing amazon for a while, though they may be catching up, thanks to comcast...
But if you want to roll your own, then you can, it's FOSS after all. kubeadm works very well.
Comparing Synology with anything Google is silly, but useful. You can rent dedicated machines easily. Leaseweb is nice. And then you can build your infrastructure for cheaper.
Yes, different things are different levels of complexity, and using a managed service means paying someone else to handle it.
Kubernetes is not going to be simpler or easier than the software that it's designed to run on top of itself, but it's not that hard anymore either. The installers work well, there are several distros with varying capabilities, or you can just use the public clouds like most where it's 1 button.