Re:Good and Bad Monitoring
by cassianoleal on Tuesday July 13, 2021 @05:34PM
(#27826585)
> Good: Alerts on business level metrics
I agree partially. I would make it more general though: alert on symptoms of problems. Those can be business metrics, like the one you suggested in your example, or they can be system level, like rate of errors, or a queue that's growing out of control.
> Bad: Alerts on machine level metrics
100% agree. There is no excuse for that. A CPU working overtime with no customer impact (no symptoms of problems) is an efficient system. I'm paying for that CPU, I'd like to use it. If I get an alert every time I use something I pay for that will only drive me to pay more so it shuts up - even though there was no problem to begin with.
I agree partially. I would make it more general though: alert on symptoms of problems. Those can be business metrics, like the one you suggested in your example, or they can be system level, like rate of errors, or a queue that's growing out of control.
> Bad: Alerts on machine level metrics
100% agree. There is no excuse for that. A CPU working overtime with no customer impact (no symptoms of problems) is an efficient system. I'm paying for that CPU, I'd like to use it. If I get an alert every time I use something I pay for that will only drive me to pay more so it shuts up - even though there was no problem to begin with.