News for hackers. Stuff that matters.
Re:Building a Fly.io-Like Scheduler with Resource Requirements by mbreese on Sunday February 25, 2024 @12:12PM (#39502535)
Another place to look for similar workflows is the HPC world. HPC schedulers have been doing resource requirement based scheduling for forever. Many of those those keep the scheduling state centralized though. Instead of asking the workers “who can fulfill these requirements”, the coordinator process keeps track of the state of what resources are still available for each worker. If you have a task that needs 12GB of RAM, 4 processors, and is in us-west, the coordinator already knows the answer. So instead of N^2 messages, you just have two (submit work and ack).

The major reason why HPC keeps this centrally is that it makes time based scheduling easier too. In general, you have more work than nodes and each job only has a finite run time. This lets the scheduler do fun things like backfilling jobs and more coordinated scheduling (such as keeping an entire rack idle for power savings unless necessary).

Re:Building a Fly.io-Like Scheduler with Resource Requirements by dangoodmanUT on Sunday February 25, 2024 @03:09PM (#39504265)
Yeah, the problem with central is that it can get out of sync, but for tracking time it's definitely superior
Re:Building a Fly.io-Like Scheduler with Resource Requirements by mbreese on Sunday February 25, 2024 @08:48PM (#39506837)
True… state can get out of sync. But in general, I think this can be (and is?) handled with a heartbeat process. The compute nodes notify the central server of their status every X minutes (or get polled by the central node). Or even when the central server sends a job out, the state of the compute node could be queried as opposed to just assumed.

In general, you’re going to have something like this running anyway just to keep track of nodes that die unexpectedly.

Re:Building a Fly.io-Like Scheduler with Resource Requirements by gregod on Sunday February 25, 2024 @06:33PM (#39506084)
Slurm [1] is a open-source HPC scheduler thats a good example. Its featureful and (surprisingly) easy to run if no user accounting (i.e., accounting for individual computation budgets) is needed.

[1]: https://slurm.schedmd.com/documentation.html

Re:Building a Fly.io-Like Scheduler with Resource Requirements by mbreese on Sunday February 25, 2024 @08:44PM (#39506811)
It’s also really good for when accounting is needed too, it’s just more complicated to setup. :)
Re:Building a Fly.io-Like Scheduler with Resource Requirements by gregod on Monday February 26, 2024 @04:00PM (#39516730)
Yes, that was poorly worded. It has a very neat accounting system! Its just a little more complex to setup and keep in sync.