News for hackers. Stuff that matters.
Re:Roblox October Outage Postmortem by roughly on Thursday January 20, 2022 @05:10PM (#30015865)
During a recent AWS outage, the STS service running in us-east-1 was unavailable. Unfortunately, all of the other _regions_ - not AZs, but _regions_, rely on the STS service in us-east-1, which meant that customers which had built around Amazon’s published reliability model had services in every region impacted by an outage in one specific availability zone.

This is what kreeben was referring to - not some abstract misconception about the difference between AZs and Regions, but an actual real world incident in which a failure in one AZ had an impact in other Regions.

Re:Roblox October Outage Postmortem by otterley on Thursday January 20, 2022 @05:35PM (#30016244)
It's more subtle than that.

For high availability, STS offers regional endpoints -- and AWS recommends using them[1] -- but the SDKs don't use them by default. The author of the client code, or the person configuring the software, has to enable them.

[1] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti...

(I work for AWS. Opinions are my own and not necessarily those of my employer.)

Re:Roblox October Outage Postmortem by roughly on Thursday January 20, 2022 @06:19PM (#30016809)
The client code which defaults to STS in us-east-1 includes the AWS console website, as far as I can tell.

Real question, though - are those genuinely separate endpoints that remained up and operational during the outage? I don’t think I saw or knew a single person unaffected by this outage, so either there’s still some bleed over on the backend or knowledge of the regional STS endpoints is basically zero (which I Can believe, y’all run a big shop)

Re:Roblox October Outage Postmortem by Karrot_Kream on Thursday January 20, 2022 @07:08PM (#30017308)
My team didn't use STS but I know other teams at the company did. Those that did rely on non-us-east-1 endpoints did stay up IIRC. Our company barely use the AWS console at all and base most of our stuff around their APIs to hook into our deployment/CI processes. But I don't work at AWS so I don't know if it's true or if there was some other backend replication lag or anything else going on that was impacted by us-east-1 being down. We had some failures for some of our older services that were not properly sharded out, but most of our stuff failed over and continued to work as expected.
Re:Roblox October Outage Postmortem by Karrot_Kream on Thursday January 20, 2022 @05:50PM (#30016466)
> Unfortunately, all of the other _regions_ - not AZs, but _regions_, rely on the STS service in us-east-1, which meant that customers which had built around Amazon’s published reliability model had services in every region impacted by an outage in one specific availability zone.

That's not true. STS offers regional endpoints, for example if you're in Australia and don't want to pay the latency cost to transit to us-east-1 [1]. It's up to the user to opt into them though. And that goes back to what I was saying earlier, you need engineers willing to read their docs closely and architect systems properly.

[1]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti...