News for hackers. Stuff that matters.
Re:Grafana OnCall: an easy-to-use on-call management tool by CSDude on Tuesday November 09, 2021 @01:20PM (#29165267)
> Alerts from each integration 300 5 minutes

> Alerts from the whole team 500 5 minutes

> API requests per API key 300 5 minutes

Product looks great but those API request limits are too low, because alerts rain when you are having incidents and rate limiting all of them is harmful. That's why other products have deduplication keys / aliases so you don't miss important ones.

https://grafana.com/docs/grafana-cloud/oncall/oncall-api-ref...

Re:Grafana OnCall: an easy-to-use on-call management tool by deeblering4 on Tuesday November 09, 2021 @01:33PM (#29165468)
I'd think that receiving even 1/5th the rate limit in a 5 minute window would be disorienting enough to render alerting effectively useless.

I'd question the configuration which fires that many alerts in that time frame, and suggest improving alert aggregations and dependencies to get the number down to one or a handful of meaningful alerts.

Re:Grafana OnCall: an easy-to-use on-call management tool by curryst on Tuesday November 09, 2021 @04:06PM (#29167474)
The overhead of maintaining those configurations all the time is usually too high to be worth it considering the benefit and likelihood of reaping it.

Also, in my experience with those systems, they only make sense to use very sparingly. Your monitoring becomes extremely fragile when your aggregations and dependencies get complicated enough that "what will our alerting system do when X happens?" results in a flow chart with 18 steps.

If you aren't careful, you can end up making your aggregations less useful than the raw alerts would be.

Re:Grafana OnCall: an easy-to-use on-call management tool by rmetzler on Tuesday November 09, 2021 @04:55PM (#29168046)
It would be great to have a dependency graph or labels in the alerts, so they are easily mapped to the things that can break and are important enough to be monitored.

We just had a short outage where an editor removed the index page in the cms which is central to the site. It's stupid that this is possible but we just operate the cms while we build and operate everything around it for our customer.

I think a large part of our alerts where triggered all at once but the one thing they had in common was that the alerts all pointed to the index page in the cms. E.g. the public www alert for index, the public api alert for index, the preview www alert for index, the preview api alert for index....

Re:Grafana OnCall: an easy-to-use on-call management tool by CSDude on Wednesday November 10, 2021 @12:47AM (#29171500)
Problem is you get same alert deduplicated hundreds of time. And with those limits, you miss others.
Re:Grafana OnCall: an easy-to-use on-call management tool by dharmab on Tuesday November 09, 2021 @06:10PM (#29168834)
I was once in a job where I was solo on call for tens of thousands of cores globally and at worst we had like 2000 alerts in a week. These limits seem quite high to me.
Re:Grafana OnCall: an easy-to-use on-call management tool by CameronNemo on Tuesday November 09, 2021 @02:04PM (#29165940)
That's why other products have deduplication keys / aliases so you don't miss important ones.

Care to link to the docs? I'm interested.

Re:Grafana OnCall: an easy-to-use on-call management tool by CSDude on Tuesday November 09, 2021 @02:21PM (#29166172)
Re:Grafana OnCall: an easy-to-use on-call management tool by CameronNemo on Tuesday November 09, 2021 @04:10PM (#29167526)
Thanks for the links.

From the article:

With Grafana OnCall’s automatic grouping of alerts within Slack, you can avoid alert storms and reduce the noise your teams are exposed to during an incident.

Seems like the same feature described using different terminology.

Re:Grafana OnCall: an easy-to-use on-call management tool by EwanToo on Tuesday November 09, 2021 @05:26PM (#29168364)
The output alerts feature looks largely the same, but the input API limits are the part in question.

What happens if you get 1000 API calls about "Alert 1" and 1 API call about "Alert 2".

You want both on call's to trigger once, but will alert 2 get though?

Re:Grafana OnCall: an easy-to-use on-call management tool by named-user on Tuesday November 09, 2021 @02:02PM (#29165923)
How else do you think they are gonna make money?