Why Ubuntu? Mostly because it just makes packaging and compatibility easier. There is a plugin system which creates tarballs out of overlays... so you can add additional software into the system and the building of the tarballs can include things like apt install. These tarballs get downloaded and installed early on by a hacked init script. This way, we can do things like install minimized downloads of GPU drivers.

There are some issues with this, but overall it works ok. For now, it is just what I ended up with. Total boot download is about 100megs, which isn't too bad... machine is up and running in 60s, which is fine for this workload right now. It does suck rebooting the whole datacenter at once though. ;-)

The lbu stuff looks interesting for sure. I'm not a huge fan of the additional NFS requirement, but I assume that can be worked around. That said, I'm open to changing stuff up and optimizing things further. This is some esoteric stuff and we are hiring. =)

My side project for the last while was pretty similar to what you're describing, and as a result I've become very familiar with this "esoteric stuff". If you're hiring, I'm interested :)

Thanks. Emailed you.