(disclaimer: I work for another cloud provider)

I agree, in general, outages are almost inevitable, but global outages shouldn't occur. It suggests at least a couple of things:

1) Bad software deployments, without proper validation. A message elsewhere in this post on HN suggest that problems have been occurring for at least 5 days, which makes me think this is the most likely situation. If this is the case, presumably given this is multiple days in to the issue, rolling back isn't an option. That doesn't say good things about their testing or deployment stories, and possibly their monitoring of the product? Even if the deployment validation processes failed to catch it, you'd really hope alarming would have caught it.

or:

2) Regions aren't isolated from each other. Cross-region dependencies are bad, for all sorts of obvious reasons.

That shouldn't, but they do. S3 goes down [1]. The AWS global console goes down, right after Prime Day outages [2]. Lots of Google Cloud services go down [3, current thread]. Tens of Azure services go down hard [4].

Are software development and release processes improving to mitigate these outages? We don't know. You have to trust the marketing. Will regions ever be fully isolated? We don't know. Will AWS IAM and console ever not be global services? We don't know.

Blah blah blah "We'll do better in the future". Right. Sure. Some service credits will get handed out and everyone will forget until the next outage.

Disclaimer: Not a software engineer, but have worked in ops most of my career. You will have downtime, I assure you. It is unavoidable, even at global scale. You will never abstract and silo everything per region.

[1] https://www.theregister.co.uk/2017/03/01/aws_s3_outage/

[2] https://www.cnbc.com/2018/07/16/aws-hits-snag-after-amazon-p...

[3] https://www.cnet.com/news/google-cloud-issues-causes-outages...

[4] https://www.datacenterknowledge.com/uptime/microsoft-blames-...

Can't speak for Google, but Facebook and Salesforce chose Cells for HA.

http://highscalability.com/blog/2012/5/9/cell-architectures....

Doesn't look like it was all that helpful to Facebook (as of 1542038976). Facebook.com errors out currently.

> Facebook Platform Appears to be down

> A check of https://developers.facebook.com/status/dashboard/ returns an error and I'm unable to login with facebook to some of my mobile apps.

https://news.ycombinator.com/item?id=18434262

Look how frequent and detailed Amazon's update logs are in that first Register article. Multiple updates throughout the day going into some detail.

When I was at google, the big outages were almost always bad routing to the service - it was never that the service couldn't handle the load, and bad service instances were kind of hidden. my service did have some problems on new releases, but because we had multiple instances we could just redirect traffic to the instances we hadn't updated, so they stayed up.

The major issue is that outages are global instead of regional, effectively making it impossible to design around using the typical region/zone redundancy.

Because they sell themselves as being far more reliable than internal IT. If they weren't selling on uptime, people probably wouldn't be quite so critical of downtime.

As a technology practitioner, it is your failing if you believe them.

Let me know the next time you hear about the CIO of a fortune 500 asking his technology practitioners to validate what he read in Gartner and heard from Diane Greene.

My advice would be to find opportunities to get paid to tell people the right answer, not to implement the wrong answer against your better judgement. Hot job market right now, more jobs than talent, all that jazz.

If you're stuck implementing a suboptimal solution, that's not your fault, and not the intent of my above comment.

Lots of wisdom in this comment

The pitch from cloud vendors always includes the idea that the cloud is more reliable than any in-house shop can achieve. So the expectation is set by the vendors.