Re:Good and Bad Monitoring
by cassianoleal on Tuesday July 13, 2021 @05:34PM
(#27826585)
> Good: Alerts on business level metrics
I agree partially. I would make it more general though: alert on symptoms of problems. Those can be business metrics, like the one you suggested in your example, or they can be system level, like rate of errors, or a queue that's growing out of control.
> Bad: Alerts on machine level metrics
100% agree. There is no excuse for that. A CPU working overtime with no customer impact (no symptoms of problems) is an efficient system. I'm paying for that CPU, I'd like to use it. If I get an alert every time I use something I pay for that will only drive me to pay more so it shuts up - even though there was no problem to begin with.
Re:Good and Bad Monitoring
by Phelinofist on Tuesday July 13, 2021 @05:01PM
(#27826269)
Why not both? Scraping machine level metrics works out of the box with most agents
It's fine to scrape the metrics, but what I'm saying is don't alert on them by default until you are sure that a particular server metric is actually a good alert.
Good: Alerts on business level metrics
Bad: Alerts on machine level metrics
Knowing that checkout volume is sharply down is far more valuable than knowing that CPU on one of the checkout servers is way up. Mainly because that high CPU may have no customer effect, so it's really not all that urgent.