News for hackers. Stuff that matters.
Re:Google App Engine and Cloud Datastore Outages by tyingq on Monday November 11, 2019 @07:44AM (#21504366)
The root cause for the outage in June was bad configuration in the network control plane: https://status.cloud.google.com/incident/cloud-networking/19...
Re:Google App Engine and Cloud Datastore Outages by solidasparagus on Monday November 11, 2019 @03:28PM (#21508543)
The real root cause was building redundant systems that share a single point of automated failure, creating a massive blast radius.

> Thirdly, the software initiating maintenance events had a specific bug, allowing it to deschedule multiple independent software clusters at once, crucially even if those clusters were in different physical locations

Re:Google App Engine and Cloud Datastore Outages by fulafel on Tuesday November 12, 2019 @01:29AM (#21511980)
The idea of a single root cause is of course fiction in most failures in complex / redundant systems.