Re:Post-mortem of March 12 Google outage
by londons_explore on Thursday March 14, 2019 @08:24PM
(#19395267)
The real issue is that Bigtable's used to be in one big shared cell which could handle temporary 190% increases in resources.
Now, to increase isolation between services, everyone is off in a partition, and each individual partition is much smaller and therefore can't withstand resource spikes without falling over.
And I bet the "emergency loan" functionality for various resources still isn't automated and still has arcane requirements to meet and a delay before it kicks in. Yay - a 15 minute delay. Thats exactly what I need when my entire service is down! /s
Now, to increase isolation between services, everyone is off in a partition, and each individual partition is much smaller and therefore can't withstand resource spikes without falling over.
And I bet the "emergency loan" functionality for various resources still isn't automated and still has arcane requirements to meet and a delay before it kicks in. Yay - a 15 minute delay. Thats exactly what I need when my entire service is down! /s