From the first SRE book [1]:

   "The error budget stems from the observation that 100% is the wrong reliability target for basically everything 
   (pacemakers and anti-lock brakes being notable exceptions). 
   
   In general, for any software service or system, 100% is not 
   the right reliability target because no user can tell the 
   difference between a system being 100% available and 99.999% available. 
   There are many other systems in the path between 
   user and service (their laptop, their home WiFi, their ISP, 
   the power grid…) and those systems collectively are far less 
   than 99.999% available. Thus, the marginal difference 
   between 99.999% and 100% gets lost in the noise of other 
   unavailability, and the user receives no benefit from the 
   enormous effort required to add that last 0.001% of 
   availability.

   If 100% is the wrong reliability target for a system, what, 
   then, is the right reliability target for the system? This 
   actually isn’t a technical question at all—it’s a product question.."

[1] https://landing.google.com/sre/sre-book/chapters/introductio...

https://landing.google.com/sre/books/

Please, please stop using mono space for quotes. It’s hard to read on small screens. Just > is better.

It’s not hard. It’s actually impossible. I agree, just use >.

If I worked for YC as a HN mod I would literally spend a bit of time every day to review as many mono space-using comments I could every day and edit them to use > instead.

Or give us actual syntax for blockquotes? (> at the beginning of a line, markdown style, would be great…) Which I feel like gets used all the time in the discourse here, and for good reasons, too.

Then you would have that bit of time back, and the rest of us could stop scrolling back and forth when someone code-blocks a blockquote.

Or write a bot to do it automatically everywhere

Or fix the CSS

The monospace is, I think, intended for code, and a lot of us use it that way.

You want something separate for code blocks and block quotes here.

Why not use italics instead? How does the ">" help? You can put it at the beginning of the paragraph, but lines get re-flowed so you can't put it at the beginning of lines.

The > was the original (?) convention for email.

But seriously HN should just allow basic quotes. And, while they are at it, increase the size of voting arrows to make it possible to reliably hit them on mobile. I get it’s somewhat nice to have a site that does not constantly redesign everything, but pretending you lost the password for the server is just overdoing it in the opposite direction.

> increase the size of voting arrows to make it possible to reliably hit them on mobile

I was left satisfied when they added the unvote/undown buttons. I only miss on my first try about 10% of the time :)

It took me two attempts to upvote this comment. :)

At least I didn't accidentally Flag something today.

Not just small screens. The right part is cut off on my 17" screen.

We don't tolerate houses collapsing out of nowhere, brakes failing over the course of normal usage and planes falling out of the sky during routine flights.

But for some reason, we HAVE TO tolerate software crapping itself once a year?

I don't accept this logic. This is just a sign of how sloppy the industry has become.

This is the reason your phone becomes obsolete after 2 years, whereas your car can continue to run after multiple decades of abuse.

First: We’re not talking about “out of nowhere” or during “routine” operation. Doing better than 99.99% uptime implies robustness to even extreme, unusual situations.

Second: Air travel could be much, much cheaper if it didn’t have to be nearly 100% reliable. This would be the right trade-off to make in almost any application that doesn’t almost guarantee deaths when it fails.

We actually do tolerate it. Plenty of critical parts in your car are designed to not be 100% available even in all expected cases.

For example, plenty of higher end cars in california come with summer tires that can't be used in cold weather/ice.

Even the brakes you are talking about must be replaced every X miles (depending on how new the car is, this may be between 10k and 50k miles)

Houses are not definitely designed to be 100% available. This is in fact why they fail due to fire or earthquake or other events. The design point is not instant failure, but it's also not "100% available".

Like the SRE book says, they make a tradeoff.

I think this is a false equivalency. If we're talking about "service unavailability", planes break all the time. Houses have to be vacated because of flooding, fire, insect infestation. Brakes do fail. Just like with software, we accept a certain level of risk in exchange for cost/convenience efficiencies (e.g. we don't want our planes to fall out of the sky, but we're okay with getting stranded in phoenix for 24 hours because of a busted landing gear).

Also, brakes contribute to service unavailability. Brake pads need to be replaced on average every 50k miles, which takes the average driver 4 years. And let's say the average length of time your car is at the mechanic's to fix brakes is 3 days. That's 3 days of unavailability every 4 years just for brake pad replacements, or 99.8% availability (two nines!), just because of brake pad repairs. Add in all the other required car maintenance, and depending on the reliability of the vehicle, and you might be down into one nine territory.

Gmail going down is like your car being in the shop. It's not equivalent to a plane crashing; the equivalent there would be the entire contents and history of your Gmail account being unrecoverably deleted, and you yourself had no backups. Of course, I'd still much rather have that happen a hundred times than be in one fatal plane crash ..

Gmail seems to have 3 nines, although I couldn't find a better reference than this [1], where other services are included:

> [Google's infrastructure] delivers Gmail and other services to hundreds of millions of users with 99.978% availability and no scheduled downtime.

[1] https://support.google.com/googlecloud/answer/6056635?hl=en

PS. 99.978% availability translates as a downtime of ~ 2 hours/year total. Not bad! But it's when things break that we realize how performant and reliable they actually are.

Edits: various typos.

I'm consistently amazed how well Google and Facebook are at staying up. They're two services that I don't think I've really experienced a broad outage. Of course with Facebook's data designs, there's sometimes quirkiness as a result, but it's rarely completely off for me.

Google, I think I've only really noticed it offline once in the past 10 years or so. Not complaining at all.

Ok but like.... it takes <3 hours to change brake pads. So point taken, but numbers are more moderate than presented.

Consider that airplanes are relatively self-contained systems, whereas most of the systems we deal with in networking cross many different independent boundaries, each of which can independently fail for any number of reasons. There's more parties involved in regular operations of distributed software than in maintaining airplanes.

Which has do you know of that are constantly being worked on, and grown in size in perpetuity?

It's not a good metaphor, even if it looks intriguing at first sight.

We wouldn't tolerate those things if we all used the one plane, car and house. That's where this comparison falls down.

The e-mail equivalent of your house falling down is data loss. This is unavailability, which is more analogous to losing your keys and not being able to get in for 30 minutes.

Not how it works. If you service has that 99.999% availability and you get a single unavailability event in a year, that's already 5 minutes of downtime completely independent from all other events that users may or may not experience, there is almost no chance of overlap between them. Users definitely notice that. And worse, events are going to be even less frequent than that and ever more noticeable and on top of that you are going to underestimate actual unavailability by at least an order.

So, can we get to the level where unavailability is actually an unnoticeable noise? Yes, but definitely not the way Google does things. I'd generalize that Google is absolutely not the place to look for ideas on reliability.

I think you have a misconception on what actual reliability is for more products and services. 99.999% is a solid service, 99.99999% is a hard to achieve target for enterprise software.

To say Google is not the place to look for reliability is a pretty comical statement.

For reference, 99.999 is 5 mins of downtime/year.

99.99999 is 3 seconds of downtime/year.

As you get beyond 5 nines, environmental factors begin to dominate (if the service is networked: network unavailability, if the service is onsite, power outages and weather) any reliability inherent in the service.

I don't have a misconception. Can you name a single internet service that has an actual five nines availability? That's definitely not google search nor gmail.

I never claimed Google has 5 9s. Your claim that 99.999% for Gmail means Google isn't a place we should go to for reliability advice is comical at best, ignorant at worst.

I couldn't make that claim, because Gmail is very far from 99.999%. I merely pointed out that SRE quote is wrong, that's the quality of reliability advice they give. I generalized it, because I've seen plenty of bad reliability advices coming from Google, even burned by some of them in the past.

If you are into reliability you really shouldn't take Google seriously.

What specifically are you disagreeing with in the quote? It does not say that Google is five nines, nor does it say whether or not five nines is desirable.

The argument is basically the following:

(1) Consider the claim that getting to 100% reliability is important.

(2) If the claim were true, then it would be important to go from 99.999999999999999% availability to 100%.

(3) That is obviously not important, so the claim must be false.

(4) Since we have shown that the best target value is not 100%, it must be some number x, with x < 100%.

Google cloud spanner has an availability SLA of 5 9s if you use multiple regions [0]. Not cheap though.

[0]: https://cloud.google.com/spanner/sla

Any pointers to where to look for good reliability examples in software world?

Genuinely curious.

I suppose a good example is whatever runs in NASA's control room or (to a lesser extent?) in control towers.

Well, I would say that the complexity of complexity is exponential.

In the aerospace industry, to get that last percent of a percent, 2 completely independent implementations of everything are used. Then to get another decimal, you add 2 more implementations and a consensus algorithm. Then of course you add static/unit/api/integration/stress/fuzz test suits for each implementation. Then test the tests. Then have a human run each test as the "second implementation" of the CI system. And so on, and so on. Each new decimal "9" cost multiple time more in human resource alone.

Then take into account the "productivity loss" of all those process and you need yet more poeple to progress as fast. Adding more people to a project has a diminishing return. After a while you can spend the entire GDP of the world and you wont be able to add another availability decimal point.

>After a while you can spend the entire GDP of the world and you wont be able to add another availability decimal point.

And in a couple decades, when the world GDP is a bit higher, formal methods will become practical in real-life situations.

This work needs to happen on both sides. Format methods need to become much cheaper and more accessible.

Well, the "good enough" point is reached when the cost of improving the reliability of the service is more than people would pay for that improved reliability.