News for hackers. Stuff that matters.
Re:AWS power failure in US-EAST-1 region killed some hardwar... by jacquesm on Thursday December 23, 2021 @03:22PM (#29666048)
To forestall that reply: I did think about it before hitting the reply button. I've seen a couple of large scale DC power outages in my 35 years of IT work. The vast majority of the hardware came through just fine. But older data centers with gear that has been running uninterrupted for many years tend to have at least some hardware that simply won't come up again, due to a variety of reasons. One can be that the on-board batteries have gone bad, which nobody noticed until the power cycle. Another is that some harddrives function well as long as they keep spinning but that has worn a nasty little spot on the bearing that keeps the spindle aloft. When it spins down, depending on the orientation it may not be able to start back up again, it may even crash while spinning down. Powercycle enough gear that has been running for years and you will most likely lose at least some small fraction of it. You could do it the next day after taking care of those and everything would likely be fine.
Re:AWS power failure in US-EAST-1 region killed some hardwar... by vlovich123 on Thursday December 23, 2021 @04:06PM (#29666471)
Wouldn’t a UPS + generator failover mean that the HW never spins down in the first place? That’s how I interpreted op’s statement anyway.
Re:AWS power failure in US-EAST-1 region killed some hardwar... by acdha on Thursday December 23, 2021 @04:21PM (#29666624)
Hopefully, but those aren't perfect. Large data centers regularly test those because things can go wrong with any of the UPS, generator[1], or the distribution hardware which switches between the line and generator power. One of the longer outages I've seen was when the distribution hardware itself failed and burnt out multiple parts which the manufacturer did not have sufficient spares in our region.

AWS has a lot of very skilled professionals who are quite familiar with those issues so I'd be quite surprised if it turned out to be something that simple but you always want to have a contingency plan you've tested for what happens if core infrastructure like that fails and takes multiple days to recover.

1. One nasty example: fuel issues which aren't immediately obvious so someone doing a 5 minute test wouldn't learn that the system wasn't going to handle an outage longer than that.

Re:AWS power failure in US-EAST-1 region killed some hardwar... by jacquesm on Thursday December 23, 2021 @04:18PM (#29666594)
I've seen plenty of UPS's fail, and generator failover is that moment when everybody stands there with their fingers crossed.

None of those things are fool proof and in case of a large scale outage, especially one that lasts more than a couple of minutes there is a fair chance that you will find the limitations of some of your contingency plans.