Re:A terrible, horrible, no-good, very bad day at Slack
by perlgeek on Tuesday July 07, 2020 @03:16PM
(#23762950)
One approach is to have two different monitoring checks that could catch the same root cause.
In this case, one check could be for max age of actually deployed load balancer config, the other could be for new webapp instances getting traffic.
It's unlikely that both monitoring checks fail to trigger, and if only one of them triggers, one should investigate why the other didn't trigger.
Of course, that requires some decent amount of awareness from the on-call engineer (and/or an explicit step for that in the playbook); far from perfect.
In this case, one check could be for max age of actually deployed load balancer config, the other could be for new webapp instances getting traffic.
It's unlikely that both monitoring checks fail to trigger, and if only one of them triggers, one should investigate why the other didn't trigger.
Of course, that requires some decent amount of awareness from the on-call engineer (and/or an explicit step for that in the playbook); far from perfect.