Today is when the Amazon brain drain sent AWS down the spout

Re:Today is when the Amazon brain drain sent AWS down the spout by nijave on Monday October 20, 2025 @07:30PM (#45650721)

[09:13 AM PDT] We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. We will provide an update by 10:00 AM PDT.

[08:43 AM PDT] We have narrowed down the source of the network connectivity issues that impacted AWS Services...

[08:04 AM PDT] We continue to investigate the root cause for the network connectivity issues...

[12:11 AM PDT] <declared outage>

They claim not to have known the root cause for ~8hr

[ Parent | on HN ]

Re:Today is when the Amazon brain drain sent AWS down the spout by anon7000 on Monday October 20, 2025 @08:17PM (#45651028)

I don’t think that’s true, there was an initial Dynamo outage that was resolved in the wee hours that ultimately cascaded into the ec2 problem that lasted most of the day

[ Parent | on HN ]

Re:Today is when the Amazon brain drain sent AWS down the spout by nijave on Monday October 20, 2025 @08:27PM (#45651085)

Was the Dynamo outage separate? My take was the NLB issue was the root cause and Dynamo was a symptom which they flipped some internal switches to mitigate the impact to that dependency

[ Parent | on HN ]

Re:Today is when the Amazon brain drain sent AWS down the spout by easton on Monday October 20, 2025 @08:59PM (#45651302)

If their internal NLB monitoring can delete the A record for dynamodb that seems like a weird dependency (like, i can imagine the nlb going missing entirely can cause it to clean up via some weird orchestration, but this didn't sound like that).

[ Parent | on HN ]

Re:Today is when the Amazon brain drain sent AWS down the spout by nijave on Tuesday October 21, 2025 @09:37AM (#45655652)

I was thinking more along the lines of the NLB being in front of DNS servers and dropping resolvers

Or an NLB could also be load balancing by managing DNS records--it's not really clear what a NLB means in this context

Or there was an overload condition because of the NLB malfunctioning that caused UDP traffic to get dropped

Obviously a lot of reading between the lines is required without a detailed RCA--hopefully they release more info

[ Parent | on HN ]

Re:Today is when the Amazon brain drain sent AWS down the spout by CoopaTroopa on Monday October 20, 2025 @09:18PM (#45651435)

Sure, that timeline looks bad when you leave out the 14 updates between 12:11am PDT and 8:04am PDT.

The initial cause appears to be a a bad DNS entry that they rolled back at 2:22am PDT. They started seeing recovery with services but as reports of EC2 failures kept rolling in they found a network issue with a load balancer that was causing the issue at 8:43am.

[ Parent | on HN ]

Re:Today is when the Amazon brain drain sent AWS down the spout by znpy on Tuesday October 21, 2025 @04:03PM (#45660967)

> Sure, that timeline looks bad when you leave out the 14 updates between 12:11am PDT and 8:04am PDT.

Their 14 updates did not bring my stuff back up.

My nines are not their nines. https://rachelbythebay.com/w/2019/07/15/giant/

[ Parent | on HN ]

Re:Today is when the Amazon brain drain sent AWS down the spout by CoopaTroopa on Thursday October 23, 2025 @09:52AM (#45681830)

I didn't say they fixed everything within those 14 updates. I'm pointing out it's disingenuous to say they didn't start working on the issue until start of business when there are 14 updates of what they have found and done during that time.

[ Parent | on HN ]