Basically, you want read-only systems with separate partitions for persistent data. Have two images to boot from and overwrite just one of them when you upgrade. This is how Balena does it and it’s a pretty common setup for IoT stuff.