Customers who have invested millions of dollars into making their stack multi-region, multi-cloud, or multi-datacenter aren't going to calmly accept the excuse that "AWS Went Down" when you can't deliver the services you contractually agreed to deliver. There are industries out there where having your service casually go down a few times a year is totally unacceptable (Healthcare, Government, Finance, etc). I worked adjacent to a department that did online retail a while ago and even an hour of outage would lose us $1M+ in business.
Similar to security, the last few 9s of availability come at a heavily increasing (log) complexity / price. The cutoff will vary case by case, and I’m sure the decision on how many 9s you need is often irrational (CEO says it can never go down! People need their pet food delivered on time!).
> Customers who have invested millions of dollars into making their stack multi-region, multi-cloud, or multi-datacenter...
It sounds like the idea may be bad for your workplace, but that doesn't make it uninformed here. For the average B2C or business-to-small-business application, the customer doesn't even know what a region or datacenter is, all they know is that "the internet" isn't working and your service went down with it. These customers also don't have an SLA with guaranteed uptimes. The only thing they agreed to were the Terms and Conditions that explicitly say "no warranty, express or implied".
If you're selling to large enterprises, yeah, "AWS went down" won't cut it. But in most other cases it will.
Given (excluding us-east-1) you’re looking at maybe an hour a year on average of regional outage, sounds like best case break even on that investment?
You will primarily be judged by how much of an inconvenience the outage was to every individual.
The best you can hope for is that the local ISP gets the blame, but honestly. It can't be more than a rounding error in the end.
Someone decided to use AWS, so there is blame to go around. I'm not saying if that blame is warranted or not, just that it sounds like a valid thing to say for people who want to blame someone.
They just think it’s something they can justify- cloud is synonymous with velocity and the perception of lower staffing costs.
A little bit of unavailability is seen as a fair trade off.
This is my impression as a c-level.
"Someone decided to use AWS, so there is blame to go around."
Upper management.
If rolling your own is faster, cheaper, and more reliable (it is), then the only justification for cloud is assigning blame. But you know what you also don't get? Accolades.
I throw a little party of one here when Office 365 or Azure or AWS or whatever Google calls it's cloud products this week is down but all our staff are able to work without issue. =)
The real reason that talented engineers secretly support all of the middle management we vocally complain about.
I don't really have much to do with contracts - but my company is stating that we have up time of 99.xx%.
In terms of contract customers don't care if I have Azure/AWS or I keep my server in the box under the stairs. Yes they do due diligence and would not buy my services if I keep it in shoe box.
But then if they loose business they come to me .. I can go after Azure/AWS but I am so small they will throw some free credits and me and tell to go off.
Maybe if you are in B2C area then yeah - your customers will probably shrug and say it was M$ or Amazon if you write sad blog post with excuses.
If you have to give service credits to customers then with "one box" you have to give 100% of customers a credit. If your services are partitioned across two "shards" then one of those shards can go down, but your credits are only paid out at 50%.
Getting to this place doesn't prevent a 100% outage and it imposes complexity. This kind of design can be planned for enterprise B2B apps when the team are experienced with enterprise clients. Many B2B SaaS are tech folk with zero enterprise experience, so they have no idea of relatively simple things that can be done to enable a shift to this architecture.
Enterprise customers do care where things are hosted. They very likely have some users in the EU, or other locations, which care more about data protection and sovereignty than the average US organization. Since they are used to hosting on-prem and doing their own due diligence they will often have preferences over hosting. In industries like healthcare, you can find out what the hosting preferences are, as well as understand how the public clouds are addressing them. While not viewed as applicable by many on HN due to the focus on B2C and smaller B2B here, this is the kind of thing that can put a worse product ahead in the enterprise scenario.
Whereas if you do something other than the industry standard of AWS (or Azure/GCP) and it goes down, clearly it's your fault.
This is terrible for many reasons, but I wouldn't be surprised to hear someone has done this.
Guaranteeing isolation between all of these different moving parts is very difficult. Even if you're not directly affected by a large cloud outage, it's becoming less-and-less common that you, or your customers, are truely isolated.
As well, if your AWS-hosted service mostly exists to service AWS-hosted customers, and AWS is down, it doesn't matter if you are down. None of your customers are operational anyways. Is this a 100% acceptable solution? Of course not. But for 95% of services/SaaS out there, it really doesn't matter.
Imagine the clout of saying : "we stayed online while AWS died"
"We stayed online when GCP, AWS, and Azure go down" is a different story. On the other hand, if those three go down simultaneously, I suspect the state of the world will be such that I'm not worried about the internet.
If nothing else, with those three all down, so will most news sources be -- so even if you're up, your customers won't get to hear about it.
A nice thing about being in a big provider is when they go down a massive portion of the internet goes down, and it makes news headlines. Users are much less likely to complain about your service being down when it's clear you're just caught up in the global outage that's affecting 10 other things they use.