True… state can get out of sync. But in general, I think this can be (and is?) handled with a heartbeat process. The compute nodes notify the central server of their status every X minutes (or get polled by the central node). Or even when the central server sends a job out, the state of the compute node could be queried as opposed to just assumed.

In general, you’re going to have something like this running anyway just to keep track of nodes that die unexpectedly.