News for hackers. Stuff that matters.
Re:AWS power failure in US-EAST-1 region killed some hardwar... by kazen44 on Thursday December 23, 2021 @03:33PM (#29666140)
Getting power right in a DC takes a ton of preventative maintenance. (doing black building tests, make sure there are working load tests which occur frequently, managing UPS battery health etc).
Re:AWS power failure in US-EAST-1 region killed some hardwar... by spenczar5 on Thursday December 23, 2021 @04:41PM (#29666840)
Yes, it's hard work... but it's work AWS should be great at. Like, really great at.

I used to work at AWS. The most worked-up I ever saw Charlie Bell (the de facto head of AWS engineering) was in the weekly operations meeting, going over a postmortem which described a near-miss as several generators failed to start properly in a power interruption. In this meeting - with nearly 1,000 participants! - Charlie got increasingly irate about the fact that this had happened. For the next few weeks, details of some sort of flushing subsystem for backup generators was an every-time topic.

Sadly, Charlie left AWS and now works at Microsoft. I can't help but wonder what that operations meeting is like now without him.

Re:AWS power failure in US-EAST-1 region killed some hardwar... by Justin_K on Sunday December 26, 2021 @08:54PM (#29698235)
This is what we've been paying for and they've failed us this time.