Scaling Kubernetes to 7,500 Nodes

Re:Scaling Kubernetes to 7,500 Nodes by toomuchtodo on Monday January 25, 2021 @05:04PM (#25909566)

Condor is clunky, but still in use in high energy physics, for example (LHC CMS detector data processing).

For greenfield deployments, I would recommend Hashicorp's Nomad before Kubernetes or Condor if your per server container intent is ~1 (bare metal with a light hypervisor for orchestration), but still steer you to Kubernetes for microservices and web-based cookie cutter apps (I know many finance shops using Nomad, but Cloudflare uses it with Consul, so no hard and fast rules).

Disclosure: Worked in HPC space managing a cluster for high energy physics. I also use (free version) Nomad for personal cluster workload scheduling.

[ Parent | on HN ]

Re:Scaling Kubernetes to 7,500 Nodes by vergessenmir on Wednesday January 27, 2021 @06:57AM (#25927137)

I admit that Nomad is a fair middle ground due to its clean DSL and also because of the homogeneity of their workloads.

The team at OpenAI used the k8s api to make extensions around multi-tenancy (across teams) to saturate available allocations, task specific scheduling modifications which were not supported by the k8s scheduler.

I don't know if Nomad has this extensibility. Their plugins were around device plugins and tasks when I last looked at it.

[ Parent | on HN ]