News for hackers. Stuff that matters.
Re:AWS power failure in US-EAST-1 region killed some hardwar... by CoastalCoder on Thursday December 23, 2021 @02:38PM (#29665619)
Any idea why a power failure would cause (or reveal) hardware damage?

Leading up to Y2K, I remember concerns about spinning hard disks not being able to start up again.

And if the power is flaky with spikes and brown-outs, I understand that's a problem.

But is either of those relevant to AWS?

Re:AWS power failure in US-EAST-1 region killed some hardwar... by mgsouth on Thursday December 23, 2021 @04:56PM (#29666969)
Slamming on the brakes can be hard on the electronics. It causes large, high-speed voltage or current spikes that kill (old, tired) components before the spikes can be clamped. Can be made worse by old electolytic caps that no longer filter effectively. Especially hard on power supply diodes and mosfets.

Here's a few references at random:

https://vbn.aau.dk/ws/portalfiles/portal/108034396/ESREF2013...

https://www.infineon.com/dgdl/s30p5.pdf?fileId=5546d46253360...

https://www.sciencedirect.com/science/article/pii/S003811010...

Re:AWS power failure in US-EAST-1 region killed some hardwar... by Spivak on Thursday December 23, 2021 @05:22PM (#29667178)
Which is why if you realize that your hard drive is failing the worst thing you can do is cut the power [1]. You get your data off that box as fast as possible.

[1] Yes yes always have backups, and RAID, and replication something about not being able to count that low. Betcha most'a'ya don't have any of that on your MBP though except for a maybe a backup that lags hours behind your work.

Re:AWS power failure in US-EAST-1 region killed some hardwar... by CoastalCoder on Thursday December 23, 2021 @07:24PM (#29668305)
> Slamming on the brakes can be hard on the electronics. It causes large, high-speed voltage or current spikes that kill (old, tired) components before the spikes can be clamped.

Suppose that the "slamming on the brakes" here is that the power lines connecting a data center to the grid get physically cut. I can see how that would cause a power spike on the grid side of the cut, but would it really create a spike on the datacenter side of the cut?

Or assuming that the wires in a cable aren't cut exactly simultaneously, does it depend on the particular phases of the AC power at the time of each wire in the cable being cut?

Re:AWS power failure in US-EAST-1 region killed some hardwar... by R0b0t1 on Thursday December 23, 2021 @07:37PM (#29668430)
Any change in the state of an inductive load causes kickback. If you sever the lines running to a building, the generator side gets the most kick, but even a fully passive device gets a kick from the collapsing field in the wires servicing it.

The motors generate appreciable back EMF when suddenly unpowered. Likewise an SMPS is an inductive load. Spikes from power supplies going off is probably what killed the disks, assuming they didn't just decide not to power on. Some disks will run for 10 years but power them off once and they don't come back up.

Re:AWS power failure in US-EAST-1 region killed some hardwar... by SteveNuts on Thursday December 23, 2021 @03:13PM (#29665960)
In my experience it's common when there's a full power down of a large data center like this. In 2013 the company I was working at had an unplanned power outage which took out 200 spinning drives, and 18 servers. I think the motors in the drives are fine running at constant speed but don't have the juice to spin up from a dead stop.

I remember IBM was renting private planes to get us replacement hardware from around the country (we were a very large customer).

Re:AWS power failure in US-EAST-1 region killed some hardwar... by lamontcg on Friday December 24, 2021 @01:00PM (#29675909)
I worked at Amazon from 2001-2006 and there were a couple of datacenter-wide power outages during my time and there was always some machines that never came back. It is just a numbers game. When you have thousands of servers affected some small fraction just don't survive the forced power cycle.
Re:AWS power failure in US-EAST-1 region killed some hardwar... by jbergens on Thursday December 23, 2021 @05:15PM (#29667125)
A very long time ago I worked for a medium sized company and there was a power outage. The servers were in a room in our building but they had ups power. The problem was that the ups's were broken/malfunctioning and the servers needed to do a disk check after an unplanned reboot. It took hours before we could work again even though most drives survived. This was also before git so we could not reach the central version control system and could not commit any code. I think all code and workspaces were on servers that was not up yet.
Re:AWS power failure in US-EAST-1 region killed some hardwar... by emptybottle on Thursday December 23, 2021 @04:33PM (#29666745)
It can be a number of things, but often times it's the thermal cycle and wear on the components.

After letting devices (especially that rarely do this) cool, and then bringing them back up to temperature some percentage of components may fail or operate out of spec.

And mechanical parts have similar issue. A motor that's been happily running for months/years may not restart after being stopped.

It's also possible that a power fault itself could damage hardware, although in my experience this type of issue is far less common.

Re:AWS power failure in US-EAST-1 region killed some hardwar... by fy20 on Friday December 24, 2021 @02:14AM (#29671160)
If you think of it in these terms ICEs are an engineering marvell.
Re:AWS power failure in US-EAST-1 region killed some hardwar... by kube-system on Thursday December 23, 2021 @02:53PM (#29665769)
If I were to make some WAGs about things that could generally be described this way:

* increased failure rates of drives (or other hardware) due to thermal cycling

* cache power failures leading to corrupted data

* previously unknown failures in recovery processes (like firmware bugs that might be described as a "hardware problem")

* cooling failures leading to hardware failure

Most of these are pretty rare issues, but it becomes more probable to happen at least once when you're running freekin' us-east-1

Re:AWS power failure in US-EAST-1 region killed some hardwar... by michaelrpeskin on Thursday December 23, 2021 @03:43PM (#29666235)
This might be old lore, but I remember many years ago that someone told me that when a spinning disk has been up to temperature for a long time there's a bit of lubricant that is vaporized and evenly distributed everywhere (like the halides in your halogen headlamps). But when it stops spinning that stuff cools down and gets sticky and ends up sticking the heads to the disk so that it can't get going anymore.

Don't know the truth to that, but it seemed like a reasonable explanation at the time.

Re:AWS power failure in US-EAST-1 region killed some hardwar... by dilyevsky on Thursday December 23, 2021 @04:41PM (#29666832)
I worked for one of the largest datacenter company in the world and significant portion of boxes would never come up after being powercycled
Re:AWS power failure in US-EAST-1 region killed some hardwar... by _3u10 on Thursday December 23, 2021 @03:47PM (#29666278)
Think of it like a car with a starter wearing out, at some point you turn the engine off and it never starts up again, until you replace the starter. Similarly bad batteries, corrosion on terminals, etc. Startup is a much different workload on electronics and in the case of HDDs mechanical devices than running.

I have an old fan that will need replacing one day that takes about 15 minutes to reach full speed as the bearings are gone. Everytime I turn it off I wonder if the next time I turn it on it will be time to replace it.

Re:AWS power failure in US-EAST-1 region killed some hardwar... by wdfx on Thursday December 23, 2021 @04:33PM (#29666739)
It's just the numbers. Eventually all the hardware will fail. Some of that hardware will require a specific set of conditions to fail. Such a condition may be current inrush from being powercycled. Given the scale of a data centre and powercycling it all at once, then some will fail due to that condition.

Further to that thought, there must be hardware in that centre which is failing and being replaced all the time due to other conditions, we just won't hear about it because it's part of normal maintenance.

Re:AWS power failure in US-EAST-1 region killed some hardwar... by prpl on Thursday December 23, 2021 @06:35PM (#29667877)
I’m sure temperature variation has something to do with it in some cases. Most parts, even when cooled, are well above room temperature for years on end, a loss of power can drop them to room temp relatively quickly
Re:AWS power failure in US-EAST-1 region killed some hardwar... by dahfizz on Thursday December 23, 2021 @03:40PM (#29666209)
https://news.ycombinator.com/item?id=29666048

TLDR if electronics are near-death, a power cycle is likely to kick it over the edge.