Have you tested unexpected powerdowns durring writes? Unless SD card is specifically designed to survive this, I would guess you might get corruption anywhere on the card, despite writing to specific area of the card, due to wear leveling algorithms. It might only become an issue when the card was used for a while, so that all flash blocks were provisioned at least once.

Not 100% of course, but so far we haven't encountered any problem. We also checksum both the operating system and all downloaded data periodically so we can catch and repair any corruption. Of course this won't help if the system doesn't boot any more. So far we haven't had this problem.

In addition to the system part being read-only, we also try to minimized writes to the data partition. We're using some of the ext4 settings to make sure writes happen less frequently as we're perfectly capable of recovering from lost data after a reboot.

I actually implemented pretty much the same layout (custom kernel, A/B partitions, but one each for the OS and the player software) for a similar use case (background music player). We used AUFS for partition layering. In particular, our setup was as such:

OS: SquashFS file system (ro) -> Config partition (ro/rw) -> tmpfs partition (rw)

Player software: SquashFS file system (ro) -> Data partition (ro/rw) -> tmpfs partition (rw)

The config partition is mounted read-only, all changes are written to the tmpfs partition. We then use a script to bulk write desired files to the Config partition (i.e. remount rw, copy files, remount ro). This limits the write time to a fraction of a second (these are only small text files).

As for the data partition, we remount the partition rw when we're updating the music files and check for filesystem consistency on bootup.

We hadn't had a single system fail because of the SD card in over 2 years.

Neat! We're also using a SquashFS root file system. It's only 30MB total at the moment.

Right now there isn't a way to do remote system configuration changes (through a UI, it's doable from a command line). Most changes happen through the "apps" deployed on those devices and those configuration is treated as volatile and might get lost during power outages (but then repaired later). If we decide to allow modification of the few system settings we have, I might treat them as a new A/B cycle and recreate the complete system with the changed settings on the other partition. That way I also get automated fallback if anything is misconfigured and (for example) the device fails to connect to the new WiFi.

Does this compare or relate to Ubuntu Snaps? Obviously not the RO portion, but the handling the updates et al?

Otherwise this is really good to know. I guess I don't run RPi's enough because my lazy unplugs haven't been an issue...yet.

It's not based on Ubuntu Snaps. It's a custom development. It predated Ubuntu Snaps by I think 2 years? I don't see any obvious reason to switch to Snaps now, since it's all been really reliable so far. The downloads are done using zsync. Usually a system update is less than 5MB total.

This is pretty neat. Have you released anything used to build and manage the Pi aspect of this?

I haven't. So far it's really customized for our use case. If you're interested in a smallish self contained code that is based on Go, you might have a look at https://github.com/gokrazy/gokrazy

My old employer used the same A/B scheme for upgrades, and it did indeed work great.