Re:Google Cloud region currently down due to water intrusion
by Waterluvian on Thursday April 27, 2023 @08:55PM
(#35736355)
I appreciate your insight into this. Thanks!
Re:Google Cloud region currently down due to water intrusion
by throwbigdata on Friday April 28, 2023 @08:35AM
(#35740532)
Who trains the trainers?
Re:Google Cloud region currently down due to water intrusion
by palcu on Friday April 28, 2023 @11:44AM
(#35742737)
Life and experience, if you're looking for a short answer. For example, last year we had an outage in London[0] and the folks who worked on it learnt a lot. Now, they applied the learnings in this incident.
Personally, I wasn't part this time for the actual mitigation of the overall Paris DC recovery, as I was busy with an unfortunate[0] side effect of the outage. These generate more anxiety, as being woken up at 6am and being told that nobody understands exactly why the system is acting this way is not great. But then again, we're trained for this situation and there are always at least several ways of fixing the issue.
Finally, it's worth repeating that incident management is just a part of the SRE job and after several years I've understood that it is not the most important one. The best SREs I know are not great when it comes to a huge incident. But, they're work has avoided the other 99 outages that could have appeared on the front page of Hacker News.
[0]: https://news.ycombinator.com/item?id=35734224