At that scale, a complete outage is unlikely. I have services which haven't gone down _at all_ for longer than a year. But we lose requests every now and then -- during a deploy, or due to a bug. So we've moved from a time-based view of outage to a request-based view.

This helps, too, as it lets us build out services to be more reliable in combination, rather than less reliable. With retries and fail-over, an outage in an entire region may not necessarily result in any user requests failing.

For scale, pre-pandemic our published figures claimed >100M MAU.

I find https://andrewaylett.github.io/multi-burn-rate-calculator/ helpful for visualising error rates -- largely cribbed from the project it's forked from :) but with the tweakables switched around and the time between alert and error budget exhaustion in the tooltip.

It's worth noting that we only evaluate our alerts at most once a minute.

Said OVH. Then their datacenter burned to the grounds.

Said Oracle. Then their DNS was misconfigured and their whole cloud went offline for 2 hours [1].

Shit happens, always, at all scales.

[1] https://ocistatus.oraclecloud.com/incidents/qjxllgkywysj

Edit: typo

OVH didn't suffer a complete outage. If you were relying on that single DC, then you're probably not sufficiently large for this to apply to you.

But perhaps my point wasn't clearly enough made: a claim of "100% uptime" on a service level isn't particularly _useful_ when our users still only see a 99.9% success rate.

I think the weak point is their domain name. I think cloud providers should have a second domain, with a different registrar and managed sompletely independently, so that if one is subject to a problem (hijacking, dns outage, etc) clients can fallback to the alternative domain.

That literally happened, they blogged about it recently. https://www.backblaze.com/blog/recent-outages-why-we-acceler...

You say that, but telephony used to be obsessed with uptime. These are the people who invented live patching of a running system, after all. Case in point, the #1 ESS telephone switch was designed for less than 2 hours of cumulative downtime over a 40-year service life. In practice, most achieved less than 1 hour. (How many nines is that?)

Which is to say, individual subscriber lines can have all sorts of faults, but they only affect that subscriber. Trunks between offices can fail, but they only affect a certain number of circuits. The central call processing ability, of the switch to react appropriately to lines changing state and numbers being dialed, for calls to be completed if both stations are available, is what had to meet that target.

I was skeptical when I first heard this, so next time I was in an office that still had a #1 (actually a 1A, it had been upgraded in the late 80s), I talked to the switchman for a bit, and he showed me the downtime counter. It's a mechanical thing like an odometer, and once a second when the switch is executing its main loop, it touches a register that keeps the counter from incrementing. If the processor halts or isn't processing calls for some reason, the counter starts counting.

The switch was installed in the mid-70s, and from the moment it took over for its crossbar predecessor, including an in-place processor upgrade, it had logged less than an hour. Most of that came a few seconds at a time when swapping between active processors during software upgrades, he said. At the time (this was around 2002 I think) it was slated to be replaced with a DMS100, but the replacement activity hadn't commenced yet. I don't know what sort of reliability numbers the DMS machines achieve, but they'd do well to match their predecessors.

To answer your question: 1 hour of downtime after 40 years is 5.5 "nines": 99.9997% availability.

So to the grandparent comment, 6x 9's of reliability is unrealistic to guarantee as an SLA.

If you get better, awesome, but SLA? Too unrealistic.

Depends. You're only thinking of one half of an SLA, the metric, rather than the measurement period.

What's the SLA penalty vs the extra the customer is willing to pay? If I think I can achieve 6 x 9's on a monthly basis, but probably I'll only achieve it 10 months out of 12, I can offer the customer an SLA of 6 x 9's for 100 USD per month.

My penalty can be 50 USD for failing to meet, and then I as a supplier walk away with 10 x 10 + 2 x 50 USD = 1100 USD for offering something I knew I couldn't achieve (consistently).

Indeed, in the early days of Amazon S3, they failed their SLA every single month. And I got... a 10% refund every month.

> For anyone interested, any more than about five 9s of reliability is basically impossible anyway when it comes to human intervention. As an example, 6 nines of reliability would allow you 3 seconds of unavailability a year so 250 milliseconds of unavailability a month.

Not sure what the qualifier “when it comes to human intervention” means, but if I ignore it then five nines is quite standard in certain sectors. For example phone switch SLA (back when I was in the biz) was measured in minutes per decade (as in “cumulative unavailability under 1/2 minutes per decade”). Large baseline power plants can and must run uninterrupted for decades.

Of course it’s a systems issue not a point solution.

Ah, apologies.

I realise in hindsight that I was rambling from the point of view of offering five nines for software that is inherently flaky/unreliable. Companies where developers cycle through and knowledge is lost, technical debt accumulates and systems are used almost counter to their intended purpose (ie Redis as a database)

In that sense, it's like constantly massaging applications to stay alive or at a reasonable level of service and so hence the assumption that someone will be paged and respond in order to preempt a failure or restore service during an outage.