Re:Building a Fly.io-Like Scheduler with Resource Requirements
by dangoodmanUT on Sunday February 25, 2024 @03:09PM
(#39504265)
Yeah, the problem with central is that it can get out of sync, but for tracking time it's definitely superior
Re:Building a Fly.io-Like Scheduler with Resource Requirements
by mbreese on Sunday February 25, 2024 @08:48PM
(#39506837)
True… state can get out of sync. But in general, I think this can be (and is?) handled with a heartbeat process. The compute nodes notify the central server of their status every X minutes (or get polled by the central node). Or even when the central server sends a job out, the state of the compute node could be queried as opposed to just assumed.
In general, you’re going to have something like this running anyway just to keep track of nodes that die unexpectedly.
Re:Building a Fly.io-Like Scheduler with Resource Requirements
by gregod on Sunday February 25, 2024 @06:33PM
(#39506084)
Slurm [1] is a open-source HPC scheduler thats a good example. Its featureful and (surprisingly) easy to run if no user accounting (i.e., accounting for individual computation budgets) is needed.
The major reason why HPC keeps this centrally is that it makes time based scheduling easier too. In general, you have more work than nodes and each job only has a finite run time. This lets the scheduler do fun things like backfilling jobs and more coordinated scheduling (such as keeping an entire rack idle for power savings unless necessary).