There used to be a tool called, IIRC, "firebreather" that was a multicast image broadcaster for booting clusters of systems. I did a search and couldn't find any reference to it now. An associate used it to boot hundreds or thousands of systems back in the early 2000s, it would take a minute or two.

Basically: The systems would PXE boot the bootloader (or maybe it was burned locally to the boot ROM or similar, I don't recall). Then firebreather would spray the kernels and associated boot data over multicast to all the machines at once. Any that were lagging behind or missed packets, would catch them the next time around, firebreather would spray the images a few times or something.

That's really cool. Luckily, our datacenters where these machines are all Tier 3, so the full reboot doesn't happen very often. We've only had to shut down a lot of racks once for some power management issues. It was a bit of a struggle, but things eventually booted.

The bigger issue that we faced was that dnsmasq couldn't keep up with all the UDP packets and we had to switch to Kea kind of mid-flight, which was a bit of work since it was a weekend (of course things fail on friday night) and restarting everything to get new leases (and a few with duplicate IPs) was a bit of nail biting. It all sorted itself out though and we haven't had any more issues on that front.

That sounds a bit like UDP multicast. One of those tragically under used features of the networking stack. Ripe for abuse were it exposed on the open Internet certainly, but still very cool.

Yes, it was indeed built on UDP multicast.

Can you elaborate, why you chose Ubuntu?

Alpine with lbu[0] seems like a perfect fit and looks IMHO less experimental. Also you can provide backups straight via iPXE[1].

Nonetheless, kudos for the slim ubuntu image :)

[0]: https://wiki.alpinelinux.org/wiki/Alpine_local_backup

[1]: https://wiki.alpinelinux.org/wiki/PXE_boot

Why Ubuntu? Mostly because it just makes packaging and compatibility easier. There is a plugin system which creates tarballs out of overlays... so you can add additional software into the system and the building of the tarballs can include things like apt install. These tarballs get downloaded and installed early on by a hacked init script. This way, we can do things like install minimized downloads of GPU drivers.

There are some issues with this, but overall it works ok. For now, it is just what I ended up with. Total boot download is about 100megs, which isn't too bad... machine is up and running in 60s, which is fine for this workload right now. It does suck rebooting the whole datacenter at once though. ;-)

The lbu stuff looks interesting for sure. I'm not a huge fan of the additional NFS requirement, but I assume that can be worked around. That said, I'm open to changing stuff up and optimizing things further. This is some esoteric stuff and we are hiring. =)

My side project for the last while was pretty similar to what you're describing, and as a result I've become very familiar with this "esoteric stuff". If you're hiring, I'm interested :)

Thanks. Emailed you.

I didn’t use PXE to boot diskless machines, but I do find it great for remote maintenance. I’ve used it for rebuilding servers remotely from a boot image with only a very limited KVM. Serving boot media over PXE was a life saver. Honestly, I’ll be keeping a PXE server (with an HTTP server for images) on all my future clusters. It’s just that useful, especially for disaster recovery.