I wonder to what extent this actually becomes less of a problem the more people use AWS. At this point AWS being down just feels like "the internet is down", it's hard for customers to be too mad at any company being down when all their competitors are too.

Though I guess there's still probably just lost revenue that could be captured by having better uptime, even if your competitors are down.

This seems like an interesting pendulum swing where the few companies not reliant on AWS could capture significant enough revenue by maintaining uptime during a potential busy season outage.

Maybe they'll just find themselves DDOSed by the sudden influx of visitors? As a small example, i think HN was slightly slower when FB had their outage.

That depends on them being up the rest of the time. If they have an equal number of outages as AWS, it has the risk to make them look worse (since all attention is on them when they're out).

These two outages have been incredibly anomalous, I doubt you'd get much revenue betting that they'll be a common occurrence.

Agree. Now viable alternatives exist. The nextgen cloud providers will learn from the weaknesses of incumbents and innovate.

True sad fact. I first thought it is a management problem but lately I see it is the tech bros who push for fads in the hopes of staying relevant and not asuming responsability for choices.

(Accidentally down-voted, apologies! I would upvote to fix, but can't... Update: fixed)

Agreed. Arguably, not using an existing cloud service is a red flag on any new hires. AWS being the primary, but experience using GCS or Azure are at least viable skills, even if your business is AWS-based.

But the "fad-based-development" meme is not going away any time soon. The incentives in the business are built around it (really! No one want's to work on a boring old relational database solution any more). In the old days it was 4th generation languages, RUP, XML and Function Point Analysis... today it's functional programming, SDKs, big-three cloud PaaS experience or (shudder) block-chain.

I think back to my much younger self, when I thought that technology was something to be mastered to solve real-world problems, and I laugh. Little did I know the real problem to be solved was to figure out how to solve those same-old business problems but with the technology of the season (Kubernetes, GraphQL or ML).

Omg, this needs to be on a plaque or something.

"Let's move our internal app with 50 users to k8s in the cloud." --true story

Amazing. And as long as "technological progress" sufficiently obscures the impact of such ridiculousness the more such projects will continue to occur.

It's a real shame that the collective world of technology does not properly respect the simple solutions that work.

It is almost funny the dichotomy here. Most technological people "admire" the simplicity, elegance and extensibility of the command line. But tell those same people that the best data store for the solution is a relational database and their nose crinkles up.

Yeah, after getting caught up in the hype for ten years I'm running back to proven tech that is flushed out (Java, Swing-omg it just works, wanting to try Ruby,even PHP is looking good at this point).

Every dependency scrutinized and discarded if possible.

I would probably work for free if someone setup their own on-prem cloud in Tanzu, Open shift, or Ranger and used old school proven frameworks for development.

Working in AWS has been a real shitty experience at these large companies. All the nit picky problems (of which there are thousands) get dumped on devs who are trying to deliver working software.

Today DO also went down. We could not login briefly.

Just the control panel or were your instances down as well?

Just the control panel, we couldn't login

maybe except a team at google? ;)

I have a story from only a few years ago where the finance section, and a good portion of management, of Google had no idea how poor their GAE solution was for uptime, until they tried to do business critical work using software that was hosted on GAE.

Uptime improved rather dramatically after that.

yeap. it's sloowwly getting there.

If you rely solely on east 1 maybe?

AWS doesn't follow their own advice about hosting multi-regional.

When us-east-1 is sufficiently borked the management API and IAM services in all regions tend to go down with it.

Static infrastructures usually avoid the fallout, but anyone dependent on the API or otherwise dynamically created resources often get caught in the blast regardless of region

I didn't hear any reports of that happening in the most recent outage. The console was inoperable but you could work around using regional console host names.

If you're referring to Dec 7, it absolutely did. Metrics went down nearly across the board, which also means most auto-scaling setups were non-functional. Cloudfront metrics didn't properly recover until the next day

Logging in with root credentials was not possible in any region, and even logging in with IAM creds in other regions yielded an intermittently buggy console

and as is usual with us-east-1 outages management API calls were a complete crap shoot regardless of region

There are some services that do have hard US-EAST-1 dependencies. Cloudfront, because of certificates. Route53. The control API for IAM (adding/removing roles, etc). And there's also the notion of "global endpoints" like https://sts.amazonaws.com... it's not clear why that exists, because it fails when us-east-1 does. It would be better to only have regional endpoints if the "global" ones are region-specific in reality. The endpoint thing is documented, but it's still confusing to people.

The dependency chains can bite you too. During the us-east-1 outage, a Lambda run by cron-like schedules via EventBridge was itself in an okay state, but the EventBridge events that kick it off were stuck in a queue that was released when the problem was fixed. So if your Lambda wasn't idempotent, and you ran it in another region during the outage, you ended up with problems.

It wasn't reliable. I heard of many more who weren't able to get in that way than who were, and was in the former category myself.

We didn't take any downtime, but if anything had gone wrong there would have been nothing we could do about it until IAM came back up.