Re:AWS us-east-1 outage
by alpha_squared on Tuesday December 07, 2021 @01:03PM
(#29475964)
That's a copy-paste, we got the same thing from our AWS contact. It's just enough info to confirm there's an issue, but not enough to give any indication on the scope or timeline to resolution.
Re:AWS us-east-1 outage
by alexatalktome on Tuesday December 07, 2021 @01:45PM
(#29476521)
Internally the rumor is that our CICD pipelines failed to stop bad commits to certain AWS services. This isn’t due to tests but due to actual pipelines infra failing.
We’ve been told to disable all pipelines even if we have time blockers or manual approval steps or failing tests
They finally updated the status page: https://status.aws.amazon.com/
Ahh good spot, it does seem that the AWS person I am speaking too has a few more bits other than what is shown on the page, they just messaged me the same message there, but added:
"All teams are engaged and continuing to work towards mitigation. We have confirmed the issue is due to multiple impaired network devices in the US-EAST-1 Region."
Doesn't sound like they are having a good day there!
I love how they are sharing this stuff out to some clients, but its technically under NDA.
Re:AWS us-east-1 outage
by alfalfasprout on Tuesday December 07, 2021 @05:10PM
(#29478852)
Yeah, we got updates via NDA too lol. Such a joke that a status page update is considered privileged lol.
"We have identified the root cause of the issues in the US-EAST-1 Region, which is a network issue with some network devices in that Region which is affecting multiple services, including the console but also services like S3. We are actively working towards recovery."