The IP appears to be us-west-2, so we will still be able to discuss us-east-1 outages alright!

but... the s3 buckets are in us-east-1, and postgres in ap-southeast-2. More regions better than one, for maximum impact with minimum effort.

The more regions a service is scattered over the greater the odds are that a single region outage somewhere in AWS will take down the whole service.

Consider the extreme case where your service is scattered over every AWS region: here an outage of any AWS region is guaranteed to take down your service.

Compare that to the case where your service is bound to only one region: then the odds of a single region outage taking down your entire service is reduced to 1 out of however many regions AWS has (assuming each region has an equal chance of suffering an outage).

To guard against outages, the failover service has to be scattered over entirely different regions (or, even better, on an entirely different service provider... which is probably a good idea anyway).

> The more regions a service is scattered over the greater the odds are that a single region outage somewhere in AWS will take down the whole service.

Agree. I think I should have suffixed a /s to my comment above.

> To guard against outages, the failover service has to be scattered over entirely different regions (or, even better, on an entirely different service provider... which is probably a good idea anyway).

Something, something... the greatest trick the devil (bigcloud) ever pulled...

Is there at least one case when whole region went down ;)) ?

At least once, in 2012? I remember because just two days before we had fully switched over to AWS and hadn’t done multi-region yet. We got our first downtime in 10 years and there was nothing we could do, unlike when we had the servers in a colo. We were at the mercy of Amazon. After that, we moved everything back to real physical servers.

Per the spreadsheet here https://awsmaniac.com/aws-outages/ :

There seem to have been multiple "full" outages in 2011-12 in AWS' us-east-1 region, which, granted, is the oldest AWS region and likely has a bunch of legacy stuff. By "full" outages I mean that a few core services fell over but the entire region become inaccessible due to those core failures.

Its over 10 years ago tho. Are there any RECENT full region outages ?

Im forseeing a full downtime in Frankfurt this winter tho. Germany is in really bad position when it comes to electricity.

It appears to be pointing at a bare EC2 instance, no doubt a lift-and-shift.

Even during the most extreme AWS events, my EC2 instances running dedicated servers kept seeing Internet traffic.

> Even during the most extreme AWS events, my EC2 instances running dedicated servers kept seeing Internet traffic.

You were just lucky enough not to have been affected by AWS outages, but many others were.

You can get a lot of resilience to failure on AWS, but simply spinning up a dedicated EC2 instance is not nearly enough.

If architected correctly, stuff deployed into aws stays up during all but the most extreme outages.

Of those "correctly" architected apps, most are not properly tested for the failovers and won't actually work as architected (because of your own bugs or because aws failover stuff has bugs and you can't even test it).

Eg, falls over due to steep traffic spikes caused by outages when autoscaling mechanisms get previously unseen levels of load increases and enter some yoyo oscillation pattern, whole AZ is overloaded because all the failovers from the other failing AZ triggering at once, hit circuit breakers, spin up too slowly to ever pass health checks etc. Or can't detect something becoming glacially slow but not outright failing.

See eg https://www.theverge.com/2021/12/22/22849780/amazon-aws-is-d... & https://www.theverge.com/2020/11/25/21719396/amazon-web-serv... etc (many more examples are out there)

And then where do we discuss these extreme outages?

Better to discuss the next day when it's over than seeing bunch of upset comments realtime posted.

If the AWS status wasn’t a static HTML page, I would agree.

HAM Radio

People joke but from the top of a decent mountain I can reach 45 miles into the bay area from me on a $150 2m radio in my truck with rooftop antenna.

That is so cool. If I had that (truck+radio) I'd be very tempted to scatter a few cheap relay nodes at good locations in the bay area. Monkeybrains+comcast is quite reliable.

The problem with that is the only people you have to talk to is hams.

(Ex ham)

Well same for computers & the internet :)

Fair point!

If there's an extreme AWS outage, you're pretty much stuck to in-person or POTS.

where did you discuss outages of HN's previous provider?

Less people are interested in outages of HN's previous provider.

reddit.... shudders

maybe not...

https://aws.amazon.com/solutions/case-studies/reddit-aurora-...

Twitter is nice for learning about outages...

/r/sysadmin