News for hackers. Stuff that matters.
Re:Outage post-mortem by _pmf_ on Monday January 13, 2014 @12:29AM (#7049093)
> There's no discussion of the human factors like how the recovery process went, how this issue was missed in testing, or what changes if any they think they should make to their incident response process.

Not every company is into that whiney startup blood and tears thing. Those "we() worked non-stop for the last 72 hours" often sound a bit desperate.

() And by "we", the PR people usually mean the engineers.

Re:Outage post-mortem by imbriaco on Monday January 13, 2014 @12:37AM (#7049122)
That's not at all what I was getting at. It's not about patting yourself on the back or trying to make the team look like heroes.

I was more wondering how the mechanics of their incident response processes were managed and whether they planned to make any changes as a result of the review of this incident. Technical remediations are all well and good, but organizational, cultural, and even procedural changes are often even more impactful after events like this.

For example:

Were they happy with the pace of communication during the outage? Do they think customers were updated frequently enough, too frequently, etc. Any changes planned?

How did they handle incident fatigue? Did they have to go to shifts to manage the recovery? Did they already have this planned or was it done on the fly? Do they plan to build any procedures to handle similar long-running events in the future?

Re:Outage post-mortem by chris_wot on Monday January 13, 2014 @01:49AM (#7049302)
I don't see the need for any of that. What does it really matter they had "incident fatigue"? I don't really care about their internal comms or escalation procedures. If I was a customer, I'd want to know what they are doing to mitigate a similar incident (which they answered), and an apology.

If I wanted a credit, or SLAs weren't met, then I'd talk directly to an account manager.