I have a series that explains how I use Ansible to provision and manage a pi cluster: https://www.jeffgeerling.com/blog/2020/raspberry-pi-cluster-...

And in addition, I also run another cluster that has all its code on GitHub and runs this site/wiki: https://www.pidramble.com/

I am partial to the iPXE boot idea, you may find https://netboot.xyz useful.

As for how to prepare the images, you need a ramfs image with the root filesystem as a cpio file and a kernel as a vmlinuz file. You then serve these in a basic HTTP file server that your iPXE config points to.

Generating the vmlinuz file is pretty easy given a kernel. The cpio file is trickier. netboot.xyz recommends just grabbing a standard distribution. Otherwise this reduces to the problem of creating your own.

It seems to be primarily for initially provisioning servers, and doesn't really seem to be targeted at their upkeep. There is Rundeck but it's far too resource heavy to be useful for rPi.

Shameless plug: After a lot of research in this, I came to the conclusion that I'd just have to make it myself, hence Covey (https://github.com/chabad360/covey) a project that's written in Go, and currently in an early alpha stage of development, but it works. More details are in the readme, feel free to email me (its on my GH profile) if you want more details than the readme provides.

EDIT: seems like your looking for something that's primarily for provisioning anyways. Whoops, I seem to have missed the middle paragraph...

EDIT 2: PXE booting may not be the best option for a relatively low ram machine like the Pi (unless you have the storage somewhere else, but that's its own can of worms), even the minimal image required to run k3s would be at least 400mb (I think I'm underestimating) there are ways to deal with that (using a compressed squashfs). But none will beat running directly on the SD card. I'd suggest following the sibling comment, and using overlayfs on top of the main fs to ensure that the image doesn't get tarnished.

If the concern is initial provisioning I handle that by using Ubuntu server for my RPi cluster. Ubuntu server for the RPi supports using cloud-init for initial configuration of the server.

In my case I only have 2 different configurations, controller node and worker node. When deploying a new worker node I use a single digit line count shell script to produce the cloud-init files for that node from my template. Basically the only changes between worker nodes are the host name and static IP assigned. I then copy the resulting files to the FAT partition of the boot sdcard.

Someday I might replace this with network booting and something like ansible but for now this is an easy solution and handles 99% of my particular needs.

If you plan to boot from sd cards (most common configuration), 95 percent of work will typically be the image for the first machine. You can then typically just flash the same image to however many pi sd cards you want. Beyond that, shell script/ansible/whatever can get the job done at boot time for whatever per node setup you might need - there’s tons of ways to accomplish this.

The "Self Hosted" list on awesome-baremetal would probably be interesting to look over for this:

https://github.com/alexellis/awesome-baremetal#self-hosted

I used Ansible for the cluster in the article, PXE boot seems great but the learning curve is steep enough for me to put it off for the moment.

For the common base image, I just build a normal Debian image, customize it (chroot) and flash it directly. For customization, maintenance and operations you want some sort of CM (Ansible, Salt, Chef or Puppet). There is also Packer, but it's not that easy to use for bare-metal images for some reason.

You could also do something like userdata where you have a script or playbook that runs on first boot and performs additional setup.

I've been meaning to try out PXE booting but am yet to.

If you can, I'd suggest using unified kernel images, they get rid of the "whole PXE setup", you just send one file and you've booted.

We used iTerm on OS X to open ssh sessions to each node in a cluster then broadcast identical sets of command lines to each cluster in a set of panes in a team I worked on a couple years ago.

Full disclosure: I'm a customer of RackN and have used it to provision several thousand physical machines and counting.

You can get a free license for RackN's digital rebar and use their edgelab.digital [1] project to pxe boot and build a four node raspberry pi k3s cluster in a few minutes.

[1] https://github.com/digitalrebar/edgelab

Last time i checked ipxe was still not supported on pi boards that I think have a very convoluted boot process. Maybe someone with more time and passion than me can eventually figure it out =)

Ansible works great on PIs

Yeah, if you're concern is managing a lot of machines, Salt, Puppet, Ansible, or Chef are the right tool; this isn't really a Docker thing. The PXE boot idea is interesting, and you can even do that diskless.

Thank you all for your input! I was certain there was some solution to this, as it's hardly a unique problem.