News for hackers. Stuff that matters.
Re:CenturyLink 911 outage was caused by a single network car... by bch on Sunday December 30, 2018 @11:46AM (#18789960)
Every time it comes to “root cause analysis” (quoting makes sense in this context) I think of https://www.kitchensoap.com/2012/02/10/each-necessary-but-on...
Re:CenturyLink 911 outage was caused by a single network car... by hinkley on Sunday December 30, 2018 @02:29PM (#18790849)
The quality of the outcome really depends on the participants. As they say, blaming the humans is easy. But it’s not five why’s.

There are lots of proximate reasons that a failure can avoid all of your sanity checks, but the simplest is when your organization is a little insane. I try to steer the conversation toward avoiding surprises, other times toward ergonomics (don’t rely so much on humans in the moment to do the right thing).

Often people don’t fight me too much on that, but sometimes it’s a near thing. Lots of senior devs are senior because they keep everybody else down, so blaming human error for an outage seems perfectly reasonable to them.

Re:CenturyLink 911 outage was caused by a single network car... by bch on Sunday December 30, 2018 @06:07PM (#18791881)
> The quality of the outcome really depends on the participants. As they say, blaming the humans is easy. But it’s not five why’s.

Agreed. I think it’s a case for running a Joy Driven Development operation first, and looking to organizational responsibility first (versus looking for somewhere to cast blame), but practically, that probably too utopian to bear out in practice. No reason not to keep it in mind, though.