Re:A terrible, horrible, no-good, very bad day at Slack
by nickromano on Tuesday July 07, 2020 @03:07AM
(#23756525)
If you use something like CloudWatch, you can set an alarm for “insufficient data” for when a monitor stops reporting.
[ Parent
| on HN ]
Re:A terrible, horrible, no-good, very bad day at Slack
by harpratap on Tuesday July 07, 2020 @04:14AM
(#23756855)
I don't think it would have been enough for this exact situation. The solution you are providing is for cases where x==nil but in their case there was never an x, so you cannot do x==nil checks unless you deliberately set up x.
Re:A terrible, horrible, no-good, very bad day at Slack
by donavanm on Tuesday July 07, 2020 @07:07AM
(#23757910)
Not quite. Cloudwatch provides for both behaviors. You can treat periods with no datapoints as “breaching” the threshold criteria. You can also independently alert for “insufficient data” where the source is unavailable, doesnt contain enough data to evaluate, etc.
In the past I did both. Always emit a 0 datapoint every period + treat missing datapoints as breaching to discover if an application wasnt consistently emitting metrics. In addition a lower severity Insufficient Data alert was used to discover/validate when a meteic stream literally didnt exist (normally through misconfiguration of metric & alarm dimensions).
Docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
Re:A terrible, horrible, no-good, very bad day at Slack
by EdwardDiego on Tuesday July 07, 2020 @03:33AM
(#23756626)
Yep, and Grafana allows alerting on No Data states.