I think the minimum number of nodes is high because they are recommending the minimum requirements for a fault tolerant setup. It is entirely possible to install either k8s or nomad on a single node, but clearly that is not a fault tolerant situation.

IIRC both k8s and nomad rely on the Raft algorithm for consensus, and so both of them inherit the requirement of a minimum of 3 nodes from that algorithm.

If you want something to run some containers in the cloud, and you aren't concerned by the occasional failure, then there is no issue running a single Nomad (or k8s) on a single machine and later adding more if and when you require it.

Yes, I did not even consider that angle.

And yes, nomad recommends 3 - 9 nodes in prod, maybe 11 or 13. The 3 are necessary to be able to tolerate one broken/maintained node and maintain the ability to schedule workloads.

You can increase the number of tolerated nodes by increasing the number of nodes - 5 tolerate 2, 9 tolerate 3, 11 tolerate 5, 13 tolerate 6, but raft becomes slower with more nodes, because raft is a synchronous protocol.

However, it is entirely possible to run a single node nomad system, if the downtime in scheduling is fine.

PSA: In practice, choose 3 or 5 nodes. Going higher exposes system to increased communications overhead. See also: all of the literature on this.

I am currently running over 13000 nodes with a 5 server cluster. These are large boxes (90gb memory, 64 cores) but cpu usage is under 20% and memory usage is under 30%.

I'm sorry, what is the exact benefit of running multiple nodes on one piece of hardware? Just software failure resilience?

I assume they meant a 5 server control plane supporting 13k worker nodes, not that they partitioned 5 large hosts into 13k smaller ones. It's a counterpoint to GP's "raft gets slow with more servers", I think.

This is correct.

We run a Raft cluster of 5 voting members with 13,000 physical Nomad nodes mixed across OS's with a bunch of random workloads using docker, exec, raw_exec, java, and some in-house drivers. I'll clarify that (and I can't... because I can't edit my post anymore :( ).

I don't see where you see multiple nodes. I'd read the parent post as: There are 5 identically sized, beefy VMs. Each of these 5 beefy VMs run 1 instance of the nomad server. And then, there is 13k other VMs or physical systems, which run nomad in client mode. These clients connect to the server to register and get allocated allocations / task groups / practically containers or vms or binaries to run.

We're nowhere near that size, but the architecture is probably identical.