News for hackers. Stuff that matters.
Re:Google App Engine and Cloud Datastore Outages by solidasparagus on Monday November 11, 2019 @03:28PM (#21508543)
The real root cause was building redundant systems that share a single point of automated failure, creating a massive blast radius.

> Thirdly, the software initiating maintenance events had a specific bug, allowing it to deschedule multiple independent software clusters at once, crucially even if those clusters were in different physical locations

Re:Google App Engine and Cloud Datastore Outages by fulafel on Tuesday November 12, 2019 @01:29AM (#21511980)
The idea of a single root cause is of course fiction in most failures in complex / redundant systems.