Google outage – resolved

Re:Google outage – resolved by zeven7 on Monday December 14, 2020 @10:45AM (#25418813)

Several years back when I was working at Google I made a mistake that caused some of the special results in the knowledge cards to become unclickable for a small subset of queries for about an hour. As part of the postmortem I had to calculate how many people likely tried to interact with it while it was broken. It was a lot and really made me realize the magnitude of an otherwise seemingly small production failure. My boss didn't give me a hard time, just pointed me toward documentation about how to write the report. And crunching the numbers is what really made me feel the weight of it. It was a good process.

I feel for the engineer who has to calculate the cost of this bug.

[ Parent | on HN ]

Re:Google outage – resolved by robotnikman on Monday December 14, 2020 @03:15PM (#25422641)

This sounds like a good practice and hopefully something they still do. Calculating the exact numbers would definitely help cement the experience and its consequences into your mind.

[ Parent | on HN ]

Re:Google outage – resolved by zymhan on Monday December 14, 2020 @10:58AM (#25418999)

Presumably there were more failures than a single engineer could've been responsible for here.

[ Parent | on HN ]

Re:Google outage – resolved by fjcp on Monday December 14, 2020 @12:15PM (#25420083)

Its absolutely possible, the worst AWS outage was caused by one engineer running the wrong command [0].

"This past Tuesday morning Pacific Time an Amazon Web Services engineer was debugging an issue with the billing system for the company’s popular cloud storage service S3 and accidentally mistyped a command. What followed was a several hours’ long cloud outage that wreaked havoc across the internet and resulted in hundreds of millions of dollars in losses for AWS customers and others who rely on third-party services hosted by AWS."

[0] - https://www.datacenterknowledge.com/archives/2017/03/02/aws-...

[ Parent | on HN ]