News for hackers. Stuff that matters.
Re:Outage Stories: The copy and paste outage by mananaysiempre on Sunday August 14, 2022 @11:59AM (#32460298)
The way I read this, the redundancy mechanism depends on a process actually being dead when monitoring thinks it’s dead. But that’s really not the way you’re supposed to do this, is it? Maybe it’s just... resting. At the very least it seems like the allegedly dead process should be shot in the head before activating the backup one (although that’s likely quite difficult to do reliably even if the mental model is simple).
Re:Outage Stories: The copy and paste outage by kevin_nisbet on Sunday August 14, 2022 @12:25PM (#32460503)
You're correct, it was a bad design. The vendor that built this system had a number of issues in this realm that demonstrated a lack of experience in building for the reliability targets of telecom. This was just one of the easier stories to tell.

STONITH is a good design option, but I think this implementation would've still struggled. The way they were tracking processes was fragile, so the shoot the process lost track of which pid it should signal. And this was a sort of clustered system, so if you shoot the node you take out other services (think of murdering a kubernetes node to terminate one process, you're going to inflict some spillover.)