What medium might be better than Slack?
(to be clear, i'm asking, not saying Slack is good for this)
Our solution is a custom notification broker that decides whom to alert and then waits for an acknowledgement. It uses different backends including our company chat.
Not complicated at all, just 100 lines of Python code that contain the business logic.
Anything that relies on a single medium is unsuitable for anything but unimportant alerts. What if Slack goes down for 2 hour? Unlikely, but definitely possible.
This ensures that every alert is explicitly acknowledged by someone, and that unimportant alerts are quickly forgotten without wondering whether someone handled them or not.
We have different applications sending alerts, not just Nagios (because Nagios sucks at processing events as opposed to states), and it would quickly become unmanageable without some sort of middleware.
Edit: or maybe something like a blog post to describe the structural details.
(we used this in addition to, not in lieu of, the pagerduty dashboard itself, as well as mobile notification via pagerduty, and Nagio's web UI.)
> fine-grained notifications
what do you mean by this, out of curiosity?
> fine-grained notifications
A common approach is having a "alerts" channel that grabs everyones attention, even though most alerts are only relevant for a small subset of people. A company-wide #general channel is bad enough, and an #alerts channel only makes it worse.
You can totally do this with Slack by having something like PagerDuty in-between. We use a custom-written alerting broker that makes it easier to correctly handle some of the more complicated cases but it's pretty much the same idea.
It lacks everything a good alerting system has (acknowledgements, fine-grained notifications...).
Even those addresses in the stacktraces could be sensitive in other situations since they contain ASLR offsets.