Re:Tell HN: AWS appears to be down again
by tyingq on Wednesday December 22, 2021 @09:00AM
(#29649364)
There are some services that do have hard US-EAST-1 dependencies. Cloudfront, because of certificates. Route53. The control API for IAM (adding/removing roles, etc). And there's also the notion of "global endpoints" like https://sts.amazonaws.com... it's not clear why that exists, because it fails when us-east-1 does. It would be better to only have regional endpoints if the "global" ones are region-specific in reality. The endpoint thing is documented, but it's still confusing to people.
The dependency chains can bite you too. During the us-east-1 outage, a Lambda run by cron-like schedules via EventBridge was itself in an okay state, but the EventBridge events that kick it off were stuck in a queue that was released when the problem was fixed. So if your Lambda wasn't idempotent, and you ran it in another region during the outage, you ended up with problems.
The dependency chains can bite you too. During the us-east-1 outage, a Lambda run by cron-like schedules via EventBridge was itself in an okay state, but the EventBridge events that kick it off were stuck in a queue that was released when the problem was fixed. So if your Lambda wasn't idempotent, and you ran it in another region during the outage, you ended up with problems.