News for hackers. Stuff that matters.

A comment on Google App Engine and Cloud Datastore Outages.

Re:Google App Engine and Cloud Datastore Outages by estebarb on Monday November 11, 2019 @07:33AM (#21504300)
I'm really waiting for the postmortem. The first services down were networking/datastore, and some minutes later all the others started to fail. My hypothesis is that network failures prevented Paxos, a CP algorithm, to go forward, blocking writes.
[ Parent | on HN ]
Re:Google App Engine and Cloud Datastore Outages by tyingq on Monday November 11, 2019 @07:44AM (#21504366)
The root cause for the outage in June was bad configuration in the network control plane: https://status.cloud.google.com/incident/cloud-networking/19...
[ Parent | on HN ]
Re:Google App Engine and Cloud Datastore Outages by solidasparagus on Monday November 11, 2019 @03:28PM (#21508543)
The real root cause was building redundant systems that share a single point of automated failure, creating a massive blast radius.

> Thirdly, the software initiating maintenance events had a specific bug, allowing it to deschedule multiple independent software clusters at once, crucially even if those clusters were in different physical locations

[ Parent | on HN ]
Re:Google App Engine and Cloud Datastore Outages by fulafel on Tuesday November 12, 2019 @01:29AM (#21511980)
The idea of a single root cause is of course fiction in most failures in complex / redundant systems.
[ Parent | on HN ]