Right the problem is really about maintaining consistency, how do you checkpoint the entire system and all your different components with their derived values.

I used to work on a system with a distributed hierarchical object store. We had copies of the data in various geographic regions, with copies of the same or different applications running in each region connected to a local copy of the store. When you write to your local copy, the storage system would propagate changes out to the other copies.

Maintaining consistency was a constant problem. One way to handle it was to give each region running a copy of a given application it's own regionally named folders in the object store.

The problem is similar to when you have application or database clusters with nodes that lose contact with each other, so called split brain syndrome, and strategies for determining which node should become authoritative. This is where the term STONITH comes from - Shoot The Other Node In The Head.

You might not want to restart the whole system, but you do need to be able to tell subsystems that they need to refresh their state. A reboot is simply a manual implementation of that, but in a continuously running system you need to build algorithms and strategies for deciding when to do such refreshes in the system itself.