News for hackers. Stuff that matters.
Re:AWS Frankfurt incident by Symbiote on Friday June 11, 2021 @06:13AM (#27471345)
> A very small number of remaining instances and volumes that were adversely affected by the increased ambient temperatures and loss of power remain unresolved.

No noticeable smoke, but that does sound a bit closer to a fire than they planned for.

Either some system(s) not shutting down when the ambient temperature becomes high, or the ambient temperature was so hot it damaged powered-off equipment (!).

Re:AWS Frankfurt incident by brentcetinich on Friday June 11, 2021 @09:40AM (#27473172)
I have been in this situation , maintenance on cooling tower 1 engineer put it into bypass mode so he can work on it safely (in case bms decides to turn it on while he is on it.) When he left the site he forgot to take it out of bypass mode. every few days the BMS rotates the cooling towers switched off tower B and turned on tower A which was still in bypss mode… so air was blowing but the water was not circulating. the temperature graphs show the effected floors of data Center space heat up very quickly. there was a lot of permanently damaged equipment 24/7 human operators ignored many warning about ambient temperature and Low humidity assuming it was system error ( does it look hot on the cameras?? shrug ?? ) various equipment have different power down temperatures the halon did not trigger for us. some equipment did not for whatever reason shutdown when they hit the cutoff temps and cooked themselves. I suspect the Low humidity caused too much static and that’s where the damage was from..

I guess thats something hard to test for , let’s put this switch in an oven and run it to make sure it auto powers down at 80c :) every time we release a firmware.

was a problem from multiple vendors who issued bios / firmware patches. Cisco, dell, Emc, brocade.

Re:AWS Frankfurt incident by paranoidrobot on Friday June 11, 2021 @06:29AM (#27471465)
There's always a risk that equipment that was running fine, will fail when powered back on, even if not damaged otherwise by the incident.

Thermal expansion/contraction is one thing - a solder joint that was marginal might have contracted enough to actually break.

Re:AWS Frankfurt incident by kube-system on Friday June 11, 2021 @09:20AM (#27472934)
IIRC it is common to experience a lot of spinning disk failures after a cooling failure.