News for hackers. Stuff that matters.
Re:AWS power failure in US-EAST-1 region killed some hardwar... by psanford on Thursday December 23, 2021 @02:30PM (#29665526)
I would stay away from any company that thinks "we need to go multicloud" as a response to this outage. This affected a single az in a single region. If it caused you downtime or a partial outage, it means you are not fully resilient to single az failures. The correct thing is to fix your application to handle that.

If you can't handle a single az failure there is no way you are going to handle failing over across different cloud providers correctly.

Re:AWS power failure in US-EAST-1 region killed some hardwar... by nonane on Thursday December 23, 2021 @03:43PM (#29666238)
> This affected a single az in a single region. If it caused you downtime or a partial outage, it means you are not fully resilient to single az failures.

This is not true. Amazon is not being upfront about what happened here. It was simply not a single AZ failure. Our us-east-1 ELB load balancers were hosed and were unable to direct traffic to other AZs - they simply stopped working an were dropping traffic. We tried creating load balancers in different AZs and that didn't work either.

How can you be resilient to single AZ failures if load balancers stop working region wide during a single AZ outage?

Re:AWS power failure in US-EAST-1 region killed some hardwar... by acdha on Thursday December 23, 2021 @04:36PM (#29666780)
Did your TAM go into any details on that? Over a couple hundred load-balancers, the only issue we had was taking longer to register new instances and that affected only a couple of them. Running services weren't interrupted, latency remained, etc. which is what I'd expect for a single AZ failure.
Re:AWS power failure in US-EAST-1 region killed some hardwar... by dragonwriter on Thursday December 23, 2021 @03:25PM (#29666067)
> If it caused you downtime or a partial outage, it means you are not fully resilient to single az failures.

Given the number of AWS global services that have dependencies on infra in US-EAST-1 (and, from the impacts of this and other past outages, seen vulnerable to single-AZ failures in US-EAST-1) that's...less avoidable for certain regions/AZs than one might naively expect. Most clouds seem to have at least some degree of this kind of vulnerability.

Re:AWS power failure in US-EAST-1 region killed some hardwar... by CubsFan1060 on Thursday December 23, 2021 @02:37PM (#29665612)
There will, however, be a lot of executives _talking_ about going multi cloud.
Re:AWS power failure in US-EAST-1 region killed some hardwar... by luhn on Thursday December 23, 2021 @03:18PM (#29666009)
To be fair though, from what I heard AZ outage caused an EC2 API brownout, so people couldn't launch new instances in the other AZs. That put a wrench in a lot of multi-AZ architectures.

Not advocating for multi-cloud though...