STONITH is designed for critical failure so you don't end up with a split-brain situation, which is far worse than a dead node. STONITH is a good thing. The problem here is more than cluster wasn't configure to survive a catastrophic switching failure.
Re:Downtime last Saturday
by el_cuadrado on Thursday December 27, 2012 @11:49AM
(#4973906)
Yep, right - fencing solution that relies on the network to stop the service. How could that possibly fail?
But I am mildly flabbergasted by the fact that GitHub uses STONITH. The technology is as safe as open-core nuclear reactor, and it works reliably only in very simple conditions.