NVMe will allow your storage to be super fast, but it doesn't add any more compute resources. So unless your K8s cluster was all about IO, i'd say no...

I daydream about running our entire company k8s infra on a single 1U with 256 Epyc threads, a few terabytes of memory, and a bunch more of tiered storage. IMO this kind of miniaturization is amazing for container orchestration. Add a couple more, overprovision by 50%, let k8s handle the high availability, and you’ve got a datacenter in a small box.

We run single node K8s. I do it for some non-prod apps. K8s deployments are a lot more easier to organize than a bunch of docker compose deployments and its also a great way to learn K8s internals, etc.

I have nightmares of someone convincing my company to do something like that and then a having a hardware, network, or power failure.

Oh for sure. I'm happy to keep it as a daydream. Cloud providers are better at high-availability than I am.

Buy 2.

4. 2 each for 2 data centers.

And, at that point, you'll probably want an easy way to orchestrate work across your multiple nodes.

Probably so.

You probably already can; how much load are your systems under?