News for hackers. Stuff that matters.
Re:Is your SRE monitoring like a car alarm or smoke alarm? by tra3 on Tuesday April 11, 2023 @04:30PM (#35531089)
I need to sit down and go through the math again, I got lost in the middle somewhere. All I know is our alerts are way too noisy now to the point where they are useless.
Re:Is your SRE monitoring like a car alarm or smoke alarm? by sparrish on Tuesday April 11, 2023 @04:41PM (#35531249)
Alert fatigue... it's common when alerts are non-actionable and it causes a lot of downtime.
Re:Is your SRE monitoring like a car alarm or smoke alarm? by bluGill on Tuesday April 11, 2023 @05:40PM (#35531971)
We write up stories to fix them, and upper management tracks progress on completion so they are not buried in the backlog.
Re:Is your SRE monitoring like a car alarm or smoke alarm? by hevans66 on Tuesday April 11, 2023 @05:02PM (#35531518)
Yes! This. This has happened to me at least two previous companies I have worked at. Everybody sets up thresholds on every possible Datadog metric and alerts become useless. That's part of the ethos of monitoring at my current company. We only set up alerts through https://heiioncall.com/ that we are convinced you absolutely need to look at right now. Anything that is not that gets shoved to a slack channel (that I have long since muted).
Re:Is your SRE monitoring like a car alarm or smoke alarm? by nh23423fefe on Tuesday April 11, 2023 @05:13PM (#35531644)
I dunno, article doesn't seem to want me to understand. It's just another, "here's a random stats calculation you cant perform in your head, isnt the english a bad way to describe this calculation?!?!? your intuition sucks when i dont explain myself....."