News for hackers. Stuff that matters.
Re:Grafana OnCall: an easy-to-use on-call management tool by abhishekjha on Tuesday November 09, 2021 @12:40PM (#29164712)
Also what happens if pagerduty goes down?
Re:Grafana OnCall: an easy-to-use on-call management tool by julianlam on Tuesday November 09, 2021 @12:43PM (#29164771)
Ideally, the services you use should handle that (detect a non-200 and fire off a backup method like a slack webhook or email.)

In reality, probably a lot of missed downtime events, and ops sleeping peacefully I guess.

Re:Grafana OnCall: an easy-to-use on-call management tool by kevindong on Tuesday November 09, 2021 @04:32PM (#29167770)
In the year I used it, I never personally noticed it going down. Although that being said, their SLA is only 99.9% delivery in any calendar month within 5 minutes. The penalty for missing that SLA is only 10% of that month's bill.

> Once an Incident is triggered, PagerDuty will deliver the First Responder Alert within the Notification Delivery Period for 99.9% of the notifications sent by PagerDuty for the Customer during any calendar month. The “Notification Delivery Period” is five (5) minutes and it is measured as the time it takes PagerDuty to deliver a First Responder Alert to telecommunication providers in accordance with the Service configuration and Contact Information.

> ...

> If PagerDuty fails to meet the SLA set forth herein, Customer may receive a service credit. Customer will be eligible for a credit toward future fees owed to PagerDuty for the PagerDuty Service. The Service Credit is calculated as ten percent (10%) of the fees paid for or attributable to the month when the alleged SLA breach occurred.

https://www.pagerduty.com/standard-service-level-agreement/

Re:Grafana OnCall: an easy-to-use on-call management tool by vorpalhex on Tuesday November 09, 2021 @12:56PM (#29164963)
It's very rare for them to go down. I think I can remember one major outage during business hours in the last few years at which point we just switched to manual monitoring for the few hours.

If that is within your outage model, you'd probably want a redundant on-call service I suppose, even if it's just escalating to a single known email or sms group.

Re:Grafana OnCall: an easy-to-use on-call management tool by jq-r on Tuesday November 09, 2021 @02:56PM (#29166646)
Your service(s) going down and pagerduty going fully down is very unlikely to happen. Even if it does, you're probably going to get called by customer support because users never go down;)