Re:Tell HN: GitHub is down again
by kelnos on Saturday November 27, 2021 @04:18PM
(#29363638)
That might be a good idea as a last-resort measure, but if you're only finding out problems because customers are telling you about them (even indirectly through a signal like this), your monitoring and alerting is woefully inadequate.
[ Parent
| on HN ]
fun fact here that no matter how short the automated system is set up to alert, customers are faster - no one knows how is that possible
Yup, that's another reason why it pays off having a direct line of communication to at least some of your top customers.
Can't tell you how many times I've had a customer write to me, only to be receiving some automated alerts 30-40 seconds later.
And the best customers for this are the pickiest annoying ones
Generally I agree. Humans are much better at recognizing patterns than machines.
But as a counterpoint I was once working in a project where our monitoring dashboard showed an anomaly in incoming traffic. It turned out that it was an ISP problem and that we were the first ones to notice according to them.
So maybe part of the answer as to why customers are faster is that different monitoring systems are monitoring each other :)