News for hackers. Stuff that matters.
Re:Ask HN: Google Search down? by breput on Monday August 08, 2022 @10:10PM (#32393524)
https://www.sfgate.com/news/article/google-electrical-incide...

Three people in critical condition after Google data center 'electrical incident' in the Council Bluffs, Iowa datacenter (us-central region, I believe)

Re:Ask HN: Google Search down? by hoofhearted on Monday August 08, 2022 @10:42PM (#32393722)
An arc blast isn’t anything to play with! It’s an electrically charged fireball that is 4 times hotter than the sun at the point of the arc. Pretty much instant 3rd degree burns, while simultaneously getting zapped and hit with an explosive shockwave.

https://youtu.be/-iClXrd50Z8 https://youtu.be/PO6see7_ODY

Re:Ask HN: Google Search down? by shadowgovt on Monday August 08, 2022 @10:49PM (#32393770)
The physics of those things is scary.

If I recall correctly, the mitigating PPE for expecting an arc flash is reflective, because so much of the energy imparted on a victim is straight-up electromagnetic radiation. You get lightbulbed to death.

Re:Ask HN: Google Search down? by hoofhearted on Monday August 08, 2022 @11:04PM (#32393843)
Yeah an arc flash suit is rated in cal/cm². It gets to a limit where the potential energy from a high voltage piece of equipment is so great that they don’t make a suit for it because the pressure from the shockwave is enough to kill you. Pretty much any utility transformer has this much potential energy, but they design them to make it near impossible to cross phases.
Re:Ask HN: Google Search down? by dgrin91 on Monday August 08, 2022 @10:26PM (#32393603)
Seems unrelated. This happened ~10 hours ago.
Re:Ask HN: Google Search down? by prescriptivist on Monday August 08, 2022 @10:42PM (#32393719)
10 hours ago we had a flood of network failures in us-central1 and saw no GCP status changes. We blindly attempted to mitigate in various ways (freezing HPAs because we thought that we were making excessive calls to external infra and getting throttled) and it resolved itself eventually. Maybe we were at fault the entire time but not seeing this issue exposed up on the GCP dashboard is infuriating.
Re:Ask HN: Google Search down? by zxcvbn4038 on Tuesday August 09, 2022 @01:42AM (#32394554)
At AWS, if the status changes then someone somewhere gets fired, so a lot of time incidents happen without being recorded on the status board. Maybe it’s the same issue with GCP, or maybe concern for their injured peers made everyone forget to update the status. I really hope the later.
Re:Ask HN: Google Search down? by nomel on Tuesday August 09, 2022 @02:15AM (#32394751)
> At AWS, if the status changes then someone somewhere gets fired, so a lot of time incidents happen without being recorded on the status board.

This can't be true, can it? What's the reason to lie, when the lie would be so incredibly obvious?

Re:Ask HN: Google Search down? by washbrain on Tuesday August 09, 2022 @03:04AM (#32395016)
It's not true. The real answer is that execs don't want to pay the costs of slo violations. If the checkmark status green, who could say whether the service was down?
Re:Ask HN: Google Search down? by Nihmie on Tuesday August 09, 2022 @02:38AM (#32394886)
It used to be that one of the best things about working at Google was the "blameless postmortem". As long as you are able to learn from an incident and weren't attempting to look at private data, then you could write up a postmortem document and actually use that as part of a promotion packet. Google would loose a key part of its soul if they were to change that.
Re:Ask HN: Google Search down? by tnova on Tuesday August 09, 2022 @08:01AM (#32396674)
AWS has a process like that as well, it’s called a COE. GP is either misinformed or making stuff up.
Re:Ask HN: Google Search down? by lupire on Monday August 08, 2022 @10:47PM (#32393756)
Could be related if eng tried to route traffic away from Council Bluffs after the accident (shut down data center for safety inspection/repair?) and failed.
Re:Ask HN: Google Search down? by utopcell on Tuesday August 09, 2022 @01:12AM (#32394382)
not at all relevant to this outage
Re:Ask HN: Google Search down? by breput on Tuesday August 09, 2022 @02:31AM (#32394852)
Do you have anything to add about the root cause?
Re:Ask HN: Google Search down? by utopcell on Tuesday August 09, 2022 @02:34AM (#32394870)
Very much doubt I am allowed to. We are racing to recover before EU reaches peak traffic.