An update on Sunday’s service disruption

Re:An update on Sunday’s service disruption by inopinatus on Tuesday June 04, 2019 @02:15AM (#20092386)

AWS's post outrage summaries are pretty much the gold standard.

e.g.

https://aws.amazon.com/message/2329B7/

https://aws.amazon.com/message/41926/

To be fair to Google, they haven't had enough time to perform a detailed autopsy, and some GCP incident summaries have shown meat on the bones e.g. https://status.cloud.google.com/incident/compute/16007. And balancing the scales, the AWS status page is notorious for showing green when things are ... not so verdant.

I have seen full <public cloud> internal outage tickets and the volume of detail is unsurprisingly vast, and boiling it down into summaries - both internal and external - without whitewashing, without emotion, to capture an honest and coherent narration of all the relevant events and all the useful forward learnings is an epic task for even a skilled technical writer and/or principal engineer. You don't get to rest just because services are up, some folks at Google will have a sleep deficit this week.

[ Parent | on HN ]

Re:An update on Sunday’s service disruption by theevilsharpie on Tuesday June 04, 2019 @02:25AM (#20092431)

Google also posts detailed postmortems for their more significant outages.

Some examples:

https://status.cloud.google.com/incident/cloud-networking/18...

https://status.cloud.google.com/incident/cloud-pubsub/19001

https://status.cloud.google.com/incident/cloud-networking/18...

https://status.cloud.google.com/incident/compute/18012

Given that this was a multi-region outage that lasted several hours and impacted a substantial number of services, I'd expect a detailed postmortem to follow.

[ Parent | on HN ]

Re:An update on Sunday’s service disruption by dmix on Tuesday June 04, 2019 @03:05AM (#20092590)

I hope one of the things Google learns in the post mortem is that the next day summary should clearly include that a full post mortem is coming in the next few days or however long.

Half the people in this thread are overlooking that fact and going into outrage mode.

[ Parent | on HN ]

Re:An update on Sunday’s service disruption by kenhwang on Tuesday June 04, 2019 @04:35AM (#20093090)

I love reading the AWS post-mortems since they're always very detailed in describing the roles of the impacted systems, the intention of the action that caused the outage, the actual action triggered, all the nuances involved in the bug or irregularities from expected behavior, impact to systems, complications, and resolution. It paints a very complex and through picture of how their massive outages are a collection of generally simple failures or oversights that had to all line up just right for catastrophic failure.

Every time I read a Google post-mortem, they seem to hand wave everything away as "a configuration error", "bug", or "bad deploy" and their resolution always has the generic "implement changes to things" that says absolutely nothing. Honestly, when the the causes of these massive disruptions are so simply dismissed, it portrays their system as frail amateur work.

[ Parent | on HN ]

Re:An update on Sunday’s service disruption by EGreg on Tuesday June 04, 2019 @02:30AM (#20092447)

“post outrage summaries” hehe

[ Parent | on HN ]