Re:Google App Engine and Cloud Datastore Outages
by solidasparagus on Monday November 11, 2019 @03:28PM
(#21508543)
The real root cause was building redundant systems that share a single point of automated failure, creating a massive blast radius.
> Thirdly, the software initiating maintenance events had a specific bug, allowing it to deschedule multiple independent software clusters at once, crucially even if those clusters were in different physical locations
> Thirdly, the software initiating maintenance events had a specific bug, allowing it to deschedule multiple independent software clusters at once, crucially even if those clusters were in different physical locations