News for hackers. Stuff that matters.
Re:My Philosophy on Alerting: Observations of a Site Reliabi... by 0xdeadbeefbabe on Monday October 13, 2014 @06:49PM (#8451098)
So I guess the author uses a smart phone as a pager, but given his passion for uptime, reliability, latency etc. I wonder if he has experimented with an actual pager.
Re:My Philosophy on Alerting: Observations of a Site Reliabi... by apposite on Monday October 13, 2014 @08:31PM (#8451343)
We use a variety of escalation techniques. As mentioned down thread, pagers are actually very unreliable. Some SREs carry pagers and mobiles. Most SREs carry phones with escalation via SMS, an actual telephone call from an automated system (after a delay), and/or escalation via a network connection over the data network. Phone calls are way way more reliable than pagers.

Unacknowledged pages escalate to a secondary oncaller (e.g. if the oncall is out of range/in a tunnel, under a bus) and tertiary depending on configuration (and then it loops, or falls to another rotation, again depending on configuration). The code and services that do escalations is deliberately and carefully vetted to have minimal overlap with production systems (who's failure they might be alerting people to).

Re:My Philosophy on Alerting: Observations of a Site Reliabi... by shackattack on Monday October 13, 2014 @10:33PM (#8451676)
This sort of mirrors how we do things at PagerDuty. Phone may be more reliable than SMS, but every telephony/messaging gateway fails sometimes. We use something like a dozen different phone/SMS gateways to prevent single points of failure, and do end-to-end testing of our SMS providers to check their uptime and latency.

Responders can customize their notification methods (push, SMS, phone, and email) and rules, so you can do things like get a lightweight push notification when an alert happens, and then a phone call 2 minutes later if you haven't acknowledged the incident. Teams get escalation timeouts that forward alerts up the chain if the primary hasn't responded after a period of time.

Re:My Philosophy on Alerting: Observations of a Site Reliabi... by dsr_ on Monday October 13, 2014 @07:12PM (#8451167)
It turns out that pagers are often unreliable; while I've never had a message not get through eventually, I have frequently seen messages get delayed for hours or even until the pager was power-cycled.

On the other hand, if you set up an IMAPS account for each person's alert address, and then have their smartphone use that account to check email, reliability is quite high. (Strangely, sysadmins tend to have wifi in their houses and good data plans, especially when the company pays for it.)

Re:My Philosophy on Alerting: Observations of a Site Reliabi... by 0xdeadbeefbabe on Monday October 13, 2014 @07:14PM (#8451177)
Ahhh I thought the medical profession had it figured out, but these unreliable pagers could be why the doctor is always late for the delivery.