A Null Pointer Exception Brought Down Mighty Google;7 Hours of Downtime

Re:A Null Pointer Exception Brought Down Mighty Google;7 Hou... by gnat on Wednesday July 09, 2025 @05:26PM (#44514895)

Ugh. Each of these points is a classic reliability precaution – yet all were missed simultaneously. As one analyst put it, Google had “written the book on Site Reliability Engineering” but still deployed code that could not handle null inputs. In hindsight, this outage looks like a string of simple errors aligning by unfortunate chance.

Yes, that's how major outages happen. By this stage of maturity any single failure generally doesn't break things dramatically. When things go this wrong, it's ALWAYS a combination of failures: failure of recovery system, omission in detection systems, gap in automated review, oversight in ...

The vacuous gotcha language is indicative of the low quality of the whole article. As Metalnem says in comments here, see the official incident report for a better writeup and more insight. https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

[ Parent | on HN ]

Re:A Null Pointer Exception Brought Down Mighty Google;7 Hou... by jaymzcampbell on Wednesday July 09, 2025 @05:38PM (#44514977)

I really love the "Swiss Cheese model" for showing this in a very explicit way, it's easy to see how the most improbably thing could happen.

https://en.wikipedia.org/wiki/Swiss_cheese_model

[ Parent | on HN ]