News for hackers. Stuff that matters.
Re:Google cloud Europe is out by rippercushions on Wednesday April 26, 2023 @07:56AM (#35712268)
Title is incorrect, this is not a general outage. There are two separate issues:

europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf...

There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it has no impact on serving and they're rolling out a fix already. https://status.cloud.google.com/incidents/uSjFxRvKBheLA4Zr5q...

Re:Google cloud Europe is out by bsdz on Wednesday April 26, 2023 @08:14AM (#35712472)
I'm not sure if it's a separate issue but I've had trouble creating new VM instances in Google Cloud Console or listing GPU types using their CLI and I'm in europe-west-2. The ticket I was following originally got merged with the Paris flood ticket (by Google). It was working until midnight (London) last night but went down before 8am before recovering about 1h ago for me. Not sure why an outage at one regional data centre can affect services elsewhere in the zone. Perhaps it's when pooling together meta data from different data centers for listing options?
Re:Google cloud Europe is out by aristus on Wednesday April 26, 2023 @08:22AM (#35712546)
Also, consider everyone either automatically or manually trying to make up for the lost capacity in eu.
Re:Google cloud Europe is out by martius on Wednesday April 26, 2023 @09:18AM (#35713191)
Cloud Console is having issues related to the outage in europe-west9

> Customer using Cloud Console globally are unable to open and view the Compute Engine related pages like: Instance creation page Disk creation page Instance templates page Instance Groups page

https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf...

Re:Google cloud Europe is out by zerd on Wednesday April 26, 2023 @01:55PM (#35717592)
I got errors trying to open the instance group list and we don't have any resources in europe-west9.
Re:Google cloud Europe is out by mostlystatic on Wednesday April 26, 2023 @09:44AM (#35713547)
Same – was unable to create new VMs in all regions between 7:15am and 11:41am UK time. Not limited to France.
Re:Google cloud Europe is out by AlfeG on Wednesday April 26, 2023 @10:30AM (#35714212)
Every customer of affected region try to restore data/compute in other regions. It's quite known and expected issue in case of region loss.
Re:Google cloud Europe is out by theolivenbaum on Wednesday April 26, 2023 @08:27AM (#35712601)
Wait it's not DNS for a change?
Re:Google cloud Europe is out by m4jor on Wednesday April 26, 2023 @10:12AM (#35713951)
It's still DNS

Droplets Nuking Servers

Re:Google cloud Europe is out by mananaysiempre on Wednesday April 26, 2023 @10:19AM (#35714058)
What’s more obscure and less tested than figurative plumbing? Literal plumbing!
Re:Google cloud Europe is out by londons_explore on Wednesday April 26, 2023 @09:07AM (#35713072)
> There is a separate issue stopping changes to HTTP load balancers across most of GCP

Is it me, or has Google had issues with pushing changes to load balancers pretty much every few months for the past decade? Even before GCP launched, people here on HN sometimes said an outage was extended because load balancer configs couldn't be changed.

Have they not considered just redesigning their config push mechanism...

Re:Google cloud Europe is out by derefr on Wednesday April 26, 2023 @09:59AM (#35713754)
My impression is that the GCLB config for a given region's LBs is a global multitenant resource (which makes sense; there are only so many network switches serving the DC); where, to avoid that resource being constantly interrupted by config changes, all individual tenant GCLB config changes instead get thrown into a queue, where some scheduled batch process reduces over all config changes in the queue to compute a new global GCLB config state every 7–15 minutes, and pushes that, interrupting the LBs just once per interval.

(A famous speedrunning analogy: you can do everything else to remediate your app problems as fast as you like, but your LB config update arrives at a bus stop, and must wait for the next bus to come. If it just missed the previous bus, it has to wait around longer for the next one.)

It's intriguing to me that AFAIK load-balancer changes in AWS don't suffer from this problem. (Though, on the other hand, CDN changes do.)

Re:Google cloud Europe is out by re-thc on Wednesday April 26, 2023 @10:05AM (#35713843)
> It's intriguing to me that AFAIK load-balancer changes in AWS don't suffer from this problem. (Though, on the other hand, CDN changes do.)

The architecture is a lot different.

Using google means working with the load balancer in some form. It's all interconnected.

AWS is all separate parts that are stitched together thinly.

E.g. you can have a single global load balancer in Google that handles your whole infrastructure (CDN and WAF are part of LB too). There isn't an AWS equivalent. You would need a global accelerator + ALBs per region and more. WAF is tied to each ALB etc.

Re:Google cloud Europe is out by wkat4242 on Wednesday April 26, 2023 @11:19AM (#35715073)
> AWS is all separate parts that are stitched together thinly.

Yeah I always hate this when I have to work with AWS. All their services feel like they were designed by completely different companies. Every management interface looks and feels different, and there are tons of services that do almost the same thing so it's not clear which would be best to use. It's a maze to me.

Luckily I don't have to work with cloud a lot but I really prefer Azure where everything is in the same console and there isn't a lot of overlap. But cloud guys seem to hate it, not sure why.

Re:Google cloud Europe is out by d0gsg0w00f on Wednesday April 26, 2023 @12:30PM (#35716331)

    > I really prefer Azure where everything is in the same console and there isn't a lot of overlap. But cloud guys seem to hate it, not sure why.
Because Azure API's are always changing and their SDK support for non-C# is wild west.

Also, everything is a Wizard because MS doesn't want to expose the sausage factory.

Re:Google cloud Europe is out by electroly on Wednesday April 26, 2023 @11:42AM (#35715537)
AWS separates the anycast LB functionality into a separate service called AWS Global Accelerator. You do get individual anycast IP addresses with that service.
Re:Google cloud Europe is out by derefr on Wednesday April 26, 2023 @11:52AM (#35715739)
Ah, interesting; it's been a while since I played with AWS, and that service wasn't there back then. I'm guessing that allocating a new AWS Global Accelerator address takes a while?
Re:Google cloud Europe is out by electroly on Wednesday April 26, 2023 @12:00PM (#35715863)
I've only done it once (the way they have it architected, it's a "set and forget" sort of thing, your LB changes don't touch the Global Accelerator) but I do seem to recall that it took awhile to create the resource. Maybe 5-10 minutes?
Re:Google cloud Europe is out by joelrwilliams1 on Thursday April 27, 2023 @08:26AM (#35727006)
5-10 minutes is accurate for creating and rolling out changes to a Global Accelerator
Re:Google cloud Europe is out by miyuru on Wednesday April 26, 2023 @11:57AM (#35715813)
> CloudFront is anycast-routed

This is false, cloudfront uses DNS (geo & latency) based load balancing.

Re:Google cloud Europe is out by tinco on Wednesday April 26, 2023 @09:35AM (#35713430)
So this is probably too soon, thoughts and prayers for the datacenter operators and staff out there, but are they going to auction off the flooded hardware? Trying to restore a flooded Google rack sounds like a super fun project.

Anyone experience with losing an entire DC to flooding?

edit: I just Googled it (lol) and this DC has to be brand spanking new (https://cloud.google.com/blog/products/infrastructure/google...), apparently they just opened it last June. Google must be livid with the contractors who built the place for it to get flooded so soon.

Re:Google cloud Europe is out by MuffinFlavored on Wednesday April 26, 2023 @09:42AM (#35713533)
> but are they going to auction off the flooded hardware?

I wonder how many inches/feet we're talking here? The hardware on the top (unless it experienced electrical short) is most likely fine?

Re:Google cloud Europe is out by numbsafari on Wednesday April 26, 2023 @09:56AM (#35713705)
I once was a customer of a DC who's roof drainage was clogged, turning it into a lake after a couple of rain storms. It then proceeded to rain inside the DC as the roof started to leak from all the pressure.

"Servers are down, I'll head over to the DC" turned into "Um... it's raining _in the DC_. Get me some tarps and get us cut over to the backup in the office".

Ah, the glory days of running out of a single co-lo across the parking lot with our "backup site" being a former broom closet.

Re:Google cloud Europe is out by MisterTea on Wednesday April 26, 2023 @10:18AM (#35714041)
As someone who has owned two commercial flat roof buildings I cant stress enough that you MUST do inspections of your roof at least twice a year. Especially if you live in a big city. I've had backups caused by kids roofing balls and bottles, stolen purse, dead squirrel, dirty balled up diapers from the neighboring apartment building. City living for ya.
Re:Google cloud Europe is out by numbsafari on Wednesday April 26, 2023 @11:11AM (#35714920)
Yeah, I'm pretty sure in this case it was a combination of having a 4ft parapet around the entire roof, and having basically never done an inspection. Not enough drains and they were all full of leaf matter.
Re:Google cloud Europe is out by sterwill on Wednesday April 26, 2023 @06:19PM (#35720777)
Many years ago, I managed a server room with dedicated cooling on the 4th floor of a 4-story building with a flat roof. One night the temp alarms went off, and when I showed up water was dripping off my overhead Liebert unit and onto the racks.

And it wasn't even raining outside! So I grab some plastic to cover the racks and phone in emergency portable cooling as the room's AC started failing.

It turns out earlier that day, a technician performing seasonal maintenance on a boiler tank on the roof had drained the tank and refilled it. But instead of directing the water out into a proper drain, he sent it down a convenient pipe that was actually a vent from our ceiling into the boiler house. The boiler was dozens of meters from my server room, but the water followed the old steel and plaster ceiling remnants over to my computers.

And this boiler water was more exciting than rain: it came with all the dissolved minerals, metals, and preservatives computers crave! I didn't lose any computers in the racks, but it killed the Liebert's control board.

Re:Google cloud Europe is out by pjc50 on Wednesday April 26, 2023 @10:01AM (#35713780)
I'm not sure what the disk encryption story is in Google Cloud but I'd rather it didn't end up on Ebay. Mind you, "flooded" covers a wide range of possibilities and a surprisingly small amount of water ingress would trip a breaker while leaving the racks in good order.
Re:Google cloud Europe is out by antonvs on Wednesday April 26, 2023 @10:16AM (#35714018)
> a surprisingly small amount of water ingress would trip a breaker while leaving the racks in good order.

If that were the case they wouldn't be saying "There is no current ETA for recovery," and "it is expected to be an extended outage. Customers are advised to failover to other regions."

Re:Google cloud Europe is out by Xylakant on Wednesday April 26, 2023 @10:24AM (#35714130)
Starting up a cloud region after a total shutdown is likely an untested procedure with no well known timeframe, even if the hardware is ok.
Re:Google cloud Europe is out by dannyw on Wednesday April 26, 2023 @11:22AM (#35715138)
If you're in the business of being a massive cloud provider, hopefully restarting a region is not an untested procedure for you.

You could always test this in a live environment before a region becomes open to customers.

Re:Google cloud Europe is out by smueller1234 on Wednesday April 26, 2023 @12:14PM (#35716100)
You're right. It's not untested at all. It's just not instantaneous, unfortunately. :)
Re:Google cloud Europe is out by Xylakant on Wednesday April 26, 2023 @12:34PM (#35716404)
“Test in a live environment before the region becomes open to customers” is a test that’s not entirely representative for “the region had an emergency shutdown with customers on it.” And the latter is something that you can’t reliably test obviously - unless you decide to crash a whole region in live traffic.
Re:Google cloud Europe is out by CydeWeys on Wednesday April 26, 2023 @11:33AM (#35715384)
There's a lot more to a datacenter building than just the servers sitting on racks. In particular here there was a fire in the power-serving infrastructure (caused by the flood presumably). So nearly all of those servers could be totally fine, just off, but if the power distribution network in the building is literally fried, that's gonna take a long time to fix.
Re:Google cloud Europe is out by packetslave on Wednesday April 26, 2023 @02:55PM (#35718358)
Having (for example) 6 inches of water in your 115kV switch room is a small-scale problem that can cause a large-scale outage.
Re:Google cloud Europe is out by briffle on Wednesday April 26, 2023 @10:12AM (#35713956)
All data in encrypted at rest, and all hard drives are destroyed on site.
Re:Google cloud Europe is out by twistedpair on Wednesday April 26, 2023 @10:15AM (#35714002)
Better than when Planet's DC actually exploded [1].

Restoration is hard when health and safety are in question. Good luck to these ops folks <3

[1] https://www.datacenterknowledge.com/archives/2008/06/01/expl...

Re:Google cloud Europe is out by verdverm on Wednesday April 26, 2023 @10:43AM (#35714427)
I doubt they would let anyone have access to their hardware. There is a ton of proprietary stuff in there
Re:Google cloud Europe is out by Pr0Ger on Wednesday April 26, 2023 @11:02AM (#35714760)
A long time ago, one server room (located in the basement of the university building) of SPB-IX was flooded. It was a fun day for engineers whom unplugged survived equipment standing knee-deep in water

It was before dam (1) was built and floods were a huge problem in SPB

[1]: https://en.wikipedia.org/wiki/Saint_Petersburg_Dam

Re:Google cloud Europe is out by wkat4242 on Wednesday April 26, 2023 @11:21AM (#35715117)
Umm thoughts and prayers? It's not as if their house is being washed away :) They just have a busy day at work. Keeps things exciting :P
Re:Google cloud Europe is out by bushbaba on Wednesday April 26, 2023 @11:33AM (#35715387)
Likely not. It’s also not Google’s first dc flood/water intrusion causing a GCP incident.
Re:Google cloud Europe is out by milesward on Wednesday April 26, 2023 @12:07PM (#35715993)
The machines are not industry standard stuff, and they don't auction, they destroy for customer security. See here: https://www.datacenterknowledge.com/google-alphabet/robots-n...
Re:Google cloud Europe is out by discordance on Wednesday April 26, 2023 @09:56PM (#35722345)
Just the drives are destroyed. The servers themselves end up in all sorts of spots:

https://www.ebay.com/b/Google-Server/11211/bn_7023306662

Re:Google cloud Europe is out by tinco on Thursday April 27, 2023 @03:13AM (#35724365)
Those are all Google search appliances, Google sold those. They're not operated by Google themselves.
Re:Google cloud Europe is out by vivegi on Wednesday April 26, 2023 @12:18PM (#35716167)
2015 Chennai (South India) Floods. It was the flood of a century. [1]

Our DC was intact, but the building and access was cut-off. We lost the backup diesel power generators in the flooding. Of course, grid power was cut-off.

Our DC operating team managed to shutdown all the servers and racks cleanly before UPS power was completely drained. The 4 engineers and 2 security guards then swam out of the compound in chest high waters. (I am not kidding).

When the rains subsided and the flood waters receded after a couple of days, we had to plan the restart. The facility still had to be certified by health and safety, but we needed to get the datacenter back up.

A secondary operations site that would remote-connect to the DC was brought up in 1 week since we estimated the rains to potentially continue for a few more days and cause interruptions. But the critical item for the plan to work was getting a new backup power setup. We rolled in a truck-mounted diesel generator and positioned it in the highest point in the campus (also close to our building tower that had the DC) and ran power cables to it (we had to source this and it was a challenge to do it with the time crunch and the rains).

We moved staff to other cities by bus (airport was shutdown) as part of our recovery plan, but we still needed connectivity to our DC for some of the critical processes.

Long story short, it worked.

I'll never forget the experience and the scars from this war story.

[1]: https://en.wikipedia.org/wiki/2015_South_India_floods

Re:Google cloud Europe is out by vinay_ys on Wednesday April 26, 2023 @01:45PM (#35717456)
Ha, you bring back old memories. We had the largest compute footprint in India at that time in Ambattur (Chennai industrial suburb). This particular DC in question was as multi-story building and the ground-floor itself was several ft above road level and there was the huge natural lake in front. Luckily heavy rains only caused havoc to the road-side storm-drains and road traffic. And we had more than 250K liters of diesel to last us more than 24 hours and we had several tankers on standby. So we didn't have to shutdown anything. Funny thing is we had selected this site less than a year ago and had discussed the 100 year flood lines and worst case probabilities of heavy rains and flooding etc. Being well-prepared really paid off.
Re:Google cloud Europe is out by vivegi on Wednesday April 26, 2023 @11:48PM (#35723085)
Yes. It was a miracle that Ambattur did not suffer as much given the proximity to Redhills lake reservoir. Had the Water Resource Department also opened the sluice gates of the Redhills reservoir like Chembarampakkam lake during the floods and incessant rains, the situation would have been different. Given Ambattur was accessible and relatively unaffected, that was the location we brought up our alternate operating site within a week.

In any case, it is good you didn't have to go through a DC recovery during one of the worst disasters in the 21st century.

The question I keep asking in all DR planning sessions/table top exercises is - what would we do if we had a situation like what happened in Fukushima or in Chennai 2015. In both cases, flooding caused failure of backup power generators. Also, what do we do when we have all or partial resources, but are faced with a denial-of-premises situation (what I faced).

Re:Google cloud Europe is out by yla92 on Wednesday April 26, 2023 @09:42AM (#35713530)
For some reasons that might be related to the 2nd issue, even though it says resolved, I am still seeing network errors in GKE nodes, located in Singapore (asia-southeast1)

  Warning  FailedToCreateRoute      4m59s                  route_controller  Could not create route fc61a148-b428-43fa-xxxx-xxxx 10.28.167.0/24 for node gke-xxx-xxx after 16.320065487s: googleapi: Error 503: INTERNAL_ERROR - Internal error. Please try again or contact Google Support.
Any facing something similar?
Re:Google cloud Europe is out by compumike on Wednesday April 26, 2023 @10:28AM (#35714174)
Wow, “physically flooded with water somehow” and “load balancers” config propagation issue are so drastically different!

Good reminder that downtime happens for many wild reasons, and you may want to take 30 seconds and set up a free website / API monitor with Heii On-Call [1] because we would have alerted you to either of these issues if they affected your app.

Really, a simple HTTP probe provides tremendous monitoring power. I already was telling people that it covered issues at the DNS, TCP, SSL certificate, load balancer, framework, and application layers. Now I will have to add “datacenter flood” as well :P

Best wishes to everyone working on europe-west-9.

[1] https://heiioncall.com/ (I recently helped build our HTTP probe background infrastructure in Crystal)

Re:Google cloud Europe is out by antonvs on Wednesday April 26, 2023 @03:16PM (#35718615)
We just use a simple cloud function for that.
Re:Google cloud Europe is out by stall84 on Wednesday April 26, 2023 @11:41AM (#35715505)
This isn't one of the under-ocean data-centers I've seen that (at least) Microsoft had been building in the Atlantic right? (They help with cooling, obviously if under ocean)
Re:Google cloud Europe is out by terom on Wednesday April 26, 2023 @03:51PM (#35719039)
Per [1], there was a related issue affecting Cloud Console operations globally, starting from the point where the incident went regional at 23:00 PDT, and lasting until 02:00 PDT-ish. It is incorrect to say that this had zero impact elsewhere.

Sounds like some global control plane related to instance management operations started returning errors once one region failed. Or perhaps it was just the UI frontend?

[1] https://status.cloud.google.com/incidents/BWK7QzFBmfaZ4iztke...

Re:Google cloud Europe is out by gbajson on Thursday April 27, 2023 @03:42AM (#35724572)
"europe-west-9 (Paris) has been physically flooded [...], but has zero impact elsewhere."

I am afraid this is not true. We have nothing in europe-west-9, but problem in this region caused global problem with Cloud Console, which hit us, because we were not able to use it for several hours.

Snippert from https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf...:

"Cloud Console: Experienced a global outage, which has been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT."