News for hackers. Stuff that matters.
Re:AWS us-east-1 down by robrtsql on Tuesday June 13, 2023 @03:19PM (#36315484)
> No, they say to use AZ's

Using 3 AZs in us-east-1 won't save you.

I guess a demanding customer would have said 'you should have implemented disaster recovery so you could failover to us-east-2' but that's easier said than done. The more regional AWS services you adopt, the bigger the impact is. How does one recover from a regional outage if their pipeline is in that region?

Re:AWS us-east-1 down by fbellag on Tuesday June 13, 2023 @03:33PM (#36315680)
What I did once I was in the position of _having_ to provide that level of support, was to run the pipeline in a third region, different from the "prod" ones. That way, worst case you can't do deployments during the outage...

Another alternative studied was to use a thirdparty ci/cd service, outside of our network. It was discarded bc you never know where that would actually run

Re:AWS us-east-1 down by robrtsql on Tuesday June 13, 2023 @03:41PM (#36315825)
> It was discarded bc you never know where that would actually run

Yep, I considered that switching to GitHub Actions would _theoretically_ eliminate the need for disaster recovery for CI/CD (since the handling of disasters is out of your hands) but in practice their SLA is far worse than just running CodePipeline in a single region.

Re:AWS us-east-1 down by fbellag on Tuesday June 13, 2023 @06:11PM (#36318296)
Yeah, that's why we went with a third region instead. But, at the end of the day, if _only_ changes are affected for a couple of hours, that wouldn't impact the service that much
Re:AWS us-east-1 down by fnordpiglet on Tuesday June 13, 2023 @03:25PM (#36315580)
I’ve worked for several systemically important megacorps where certain things had to not only run cross region but also cross provider. It’s absurdly difficult, and only should be done if you need five or more 9’s of availability. Almost nothing actually does.