Re:Metastable Failures in Distributed Systems [pdf]
by leghifla on Tuesday October 05, 2021 @03:56AM
(#28756588)
In your post:
"There's no more time-honored way to get things working again, from toasters to global-scale distributed systems, than turning them off and on again"
This is generally true, but as all rules, there are exceptions, and I encountered one a few month ago:
To be short, the system (an embedded soft real-time control) ran fine for a long time, and the user added more and more processes. After some glitch and "to be sure the restart fresh", he initiated a reboot... And then nothing worked anymore!
The problem: each process consumed a lot of RAM for a short period at their start. When the user added processes manually, everything ran smoothly. But as soon as a few processes needed to start roughly in sync, it took too much RAM, the OOM killer killed the entire app, and back to square one.
In a way, this is also an example of metastability: the application is restarting in a loop and cannot exit that loop on its own.
This is generally true, but as all rules, there are exceptions, and I encountered one a few month ago:
To be short, the system (an embedded soft real-time control) ran fine for a long time, and the user added more and more processes. After some glitch and "to be sure the restart fresh", he initiated a reboot... And then nothing worked anymore!
The problem: each process consumed a lot of RAM for a short period at their start. When the user added processes manually, everything ran smoothly. But as soon as a few processes needed to start roughly in sync, it took too much RAM, the OOM killer killed the entire app, and back to square one.
In a way, this is also an example of metastability: the application is restarting in a loop and cannot exit that loop on its own.