Re:AWS power failure in US-EAST-1 region killed some hardwar...
by ksec on Thursday December 23, 2021 @04:20PM
(#29666607)
Yes. I expect software and network glitches because they are complicated and error prone. But I fully expect hardware and power redundancy. Even if they were down it shouldn't "kill" them. At least not on AWS, the biggest and arguably best DC running in the world. But it did.
This suggest to me DataCenter, as an infrastructure and building on itself still have plenty of room for improvements.
Re:AWS power failure in US-EAST-1 region killed some hardwar...
by kazen44 on Thursday December 23, 2021 @03:33PM
(#29666140)
Getting power right in a DC takes a ton of preventative maintenance. (doing black building tests, make sure there are working load tests which occur frequently, managing UPS battery health etc).
Re:AWS power failure in US-EAST-1 region killed some hardwar...
by spenczar5 on Thursday December 23, 2021 @04:41PM
(#29666840)
Yes, it's hard work... but it's work AWS should be great at. Like, really great at.
I used to work at AWS. The most worked-up I ever saw Charlie Bell (the de facto head of AWS engineering) was in the weekly operations meeting, going over a postmortem which described a near-miss as several generators failed to start properly in a power interruption. In this meeting - with nearly 1,000 participants! - Charlie got increasingly irate about the fact that this had happened. For the next few weeks, details of some sort of flushing subsystem for backup generators was an every-time topic.
Sadly, Charlie left AWS and now works at Microsoft. I can't help but wonder what that operations meeting is like now without him.
My 2 cents. Outages happen. Network glitches happen. Bad configs and bad updates happen. However, power issues should not really happen. One of the primary cost saving areas of going to the cloud is not having to do on-prem power, such as UPS, generators, maintenance etc. Not having to do on-prem cooling is another thing. These should be solved from the customer’s perspective when going into a professional data center and are the things you don’t want to worry about anymore.