It's all about getting them deployed effortlessly for us, and treating them as first class citizens in terms of management. Most of the devices like this you can pivot to another hardware platform if you wanted or needed.
It gets much more complicated when you start doing video (especially if you try to containerize your application, but it is possible to do while remaining performant.)
I think about 60% of our use cases boil down to "put chromium on the screen" though.
There's then a bunch of other stuff you want to do like turning down log levels, and only logging super important stuff to the SD card, which will drastically improve the lifetime of your storage.
I'm thinking of going to NixOS. The filesystem will still be mutating, but in a much more atomic way. And if a host does die, I can easily rebuild an image from another host and pave over its whole sd card.
The most robust way would be the embedded approach of two static flash partitions (update each one alternately), a separate partition for config, and another for mutable state. Alas that's more development work than I want to do for personal projects.
Just to let you know, this is the approach that balenaOS takes. Applications are containerized and deployed using Docker. The OS uses read-only system partitions to store the engine, kernel, firmware, etc, and has an A/B partition layout for resiliency and atomic updates. Applications are downloaded and stored on a separate data partition, and mutable configs are stored on another. The partition layout and a lot of how the OS functions is also well documented and public. [0]
Additionally, the first ten devices are free and fully featured, and the same people that help create the product, including (but not limited to) engineers, monitor the forums to help users. I'm one of those engineers. :)
[0]: https://www.balena.io/docs/learn/more/masterclasses/host-os-...
Neat project, but this implies some sort of ongoing third party dependency/control that the devices would have, making it a non-starter for my use. The main reason I am using RPis as security cameras (for one use) is to avoid these type of security vulnerabilities.
There's OpenBalena for hosting your own device deployment and management server. https://www.balena.io/open/
It's self-hosted, which is a double-edged sword. You have to setup and maintain your own infrastructure, but you don't depend on a third party. For personal projects, it's probably the way to go.
Additionally, devices can be moved between servers easily. If you decide to use the cloud hosted platform in the future, or move devices from cloud hosted to self-hosted, it's easy to do.