News for hackers. Stuff that matters.
Re:AWS us-east-1 outage by rehevkor5 on Tuesday December 07, 2021 @03:41PM (#29477940)
Usually this refers to falling back to a different region in AWS. It's typical for systems to be deployed in multiple regions due to latency concerns, but it's also important for resiliency. What you call "a very small edge case" is occurring as we speak, and if you're vulnerable to it you could be losing millions of dollars.
Re:AWS us-east-1 outage by simmanian on Tuesday December 07, 2021 @03:58PM (#29478138)
AWS itself has a huge single point of failure on us-east-1 region. Usually, if us-east-1 goes down, others soon follow. At that point, it doesn't matter how many regions you're deploying to.
Re:AWS us-east-1 outage by scoopertrooper on Tuesday December 07, 2021 @04:12PM (#29478296)
My workloads on Sydney and London are unaffected. I can't speak for anywhere else.
Re:AWS us-east-1 outage by lowbloodsugar on Tuesday December 07, 2021 @04:15PM (#29478320)
"Usually"? When has that ever happened?
Re:AWS us-east-1 outage by dylan604 on Tuesday December 07, 2021 @05:11PM (#29478866)
Re:AWS us-east-1 outage by lowbloodsugar on Wednesday December 08, 2021 @12:05AM (#29481443)
Thanks. That shows that OPs claim that "Usually, if us-east-1 goes down, others soon follow." is false.
Re:AWS us-east-1 outage by dijit on Wednesday December 08, 2021 @02:39AM (#29482184)
Cognito (and r53) have hard dependencies in us-east-1. It’s mentioned up and down this thread.
Re:AWS us-east-1 outage by lowbloodsugar on Wednesday December 08, 2021 @12:07PM (#29487327)
I don't want to minimize the impact from cognito and r53, but that's quite a different scale of failure than the OP implies. It's been a while since I used AWS, but we had multiple regions and saw no impact to our services in other regions the one time that us-east-1 had a major failure. And we used r53.
Re:AWS us-east-1 outage by simmanian on Wednesday December 08, 2021 @12:24PM (#29487604)
Perhaps I could've been more precise with my words. What I meant to say is IAM and r53 are two of many critical services that all depend on us-east-1. It goes without saying that if those services go down in us-east-1, the whole AWS is affected. This doesn't just happen "usually." When IAM goes down, AWS experiences major issues across all regions. If you were okay, perhaps you got lucky? Our team has 7 different prod regions and we see multiple regions go down every time a problem of this scale occurs.

If your product requires 100% uptime, you may need to look at backup options or design your product in such a way that can handle temporary cloud failures.

Re:AWS us-east-1 outage by cowmoo728 on Tuesday December 07, 2021 @06:59PM (#29479782)
I heard from someone at <big media company> that they couldn't switch to their fallback region because they couldn't update DNS on Route53. All the console commands and web interface were failing.
Re:AWS us-east-1 outage by stevehawk on Tuesday December 07, 2021 @04:06PM (#29478225)
probably not possible for a lot mroe services than you'd think because AWS Cognito has no decent failover method