Yeah I agree that false positives are a risk with monitoring 4xx rate. I've never seen a satisfactory "bulletproof" way to monitor 4xx rate, it's inherently difficult and simultaneously important to monitor.
It's easy in retrospect to say "oh that was a waste of time because it was just bots" but you don't know that until you investigate. I ask myself "if I see elevated 4xx's, at what point do I start to care if they're caused by a bug?" and set monitor thresholds somewhere around there.
The client who's calling wrong might want some help getting it right.
Its empathetic to keep an eye on surges in 4xxs
Re:Good and Bad Monitoring
by closeparen on Tuesday July 13, 2021 @09:53PM
(#27828659)
Sure, but there’s a big world between keeping an eye on surges and waking someone up if it goes out of 3-4 nines.
Re:Good and Bad Monitoring
by raynorelyp on Friday July 16, 2021 @05:48PM
(#27861624)
Author here. This.
I had something like that on an internal system. The 400 rate would jump all over the place because our edge systems had shitty input validation, and bots would crawl us with broken requests ("can I reserve this item starting last week?" kind of thing) with no rate throttling. After a few years the edge validation (and bot detection) got better, but alerting on that would've been worse than useless.