Re:Nov 16 GCP Load Balancing Incident Report
by skunkworker on Tuesday November 23, 2021 @02:02AM
(#29315295)
This, as the state continues to worsen, the higher the chance that someone observing will go "huh that looks off" and then look into it, all while your system hasn't toppled over yet, no notice or write up would be necessary, but you definitely know now what the problem is. And then following that while you are working on a patch the system finally topples over and causes an incident/outage.
Re:Nov 16 GCP Load Balancing Incident Report
by mattlondon on Tuesday November 23, 2021 @02:18AM
(#29315375)
There likely was monitoring for various "problems" in production - error rates, validation failures etc, or even just good old crash counts.
An alert may have fired that lead to someone debugging the issue in detail.
I can totally imagine a slow creeping Metric Of Death that has slowly slowly slowly been creeping up for ages and then suddenly breaches some threshold and then becomes a problem.
i would ask how the engineer found the race condition, and whether that doesn’t imply a much greater risk.