> Customer using Cloud Console globally are unable to open and view the Compute Engine related pages like: Instance creation page Disk creation page Instance templates page Instance Groups page
https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf...
Droplets Nuking Servers
Is it me, or has Google had issues with pushing changes to load balancers pretty much every few months for the past decade? Even before GCP launched, people here on HN sometimes said an outage was extended because load balancer configs couldn't be changed.
Have they not considered just redesigning their config push mechanism...
(A famous speedrunning analogy: you can do everything else to remediate your app problems as fast as you like, but your LB config update arrives at a bus stop, and must wait for the next bus to come. If it just missed the previous bus, it has to wait around longer for the next one.)
It's intriguing to me that AFAIK load-balancer changes in AWS don't suffer from this problem. (Though, on the other hand, CDN changes do.)
The architecture is a lot different.
Using google means working with the load balancer in some form. It's all interconnected.
AWS is all separate parts that are stitched together thinly.
E.g. you can have a single global load balancer in Google that handles your whole infrastructure (CDN and WAF are part of LB too). There isn't an AWS equivalent. You would need a global accelerator + ALBs per region and more. WAF is tied to each ALB etc.
Yeah I always hate this when I have to work with AWS. All their services feel like they were designed by completely different companies. Every management interface looks and feels different, and there are tons of services that do almost the same thing so it's not clear which would be best to use. It's a maze to me.
Luckily I don't have to work with cloud a lot but I really prefer Azure where everything is in the same console and there isn't a lot of overlap. But cloud guys seem to hate it, not sure why.
> I really prefer Azure where everything is in the same console and there isn't a lot of overlap. But cloud guys seem to hate it, not sure why.
Because Azure API's are always changing and their SDK support for non-C# is wild west.Also, everything is a Wizard because MS doesn't want to expose the sausage factory.
This is false, cloudfront uses DNS (geo & latency) based load balancing.
Anyone experience with losing an entire DC to flooding?
edit: I just Googled it (lol) and this DC has to be brand spanking new (https://cloud.google.com/blog/products/infrastructure/google...), apparently they just opened it last June. Google must be livid with the contractors who built the place for it to get flooded so soon.
I wonder how many inches/feet we're talking here? The hardware on the top (unless it experienced electrical short) is most likely fine?
"Servers are down, I'll head over to the DC" turned into "Um... it's raining _in the DC_. Get me some tarps and get us cut over to the backup in the office".
Ah, the glory days of running out of a single co-lo across the parking lot with our "backup site" being a former broom closet.
And it wasn't even raining outside! So I grab some plastic to cover the racks and phone in emergency portable cooling as the room's AC started failing.
It turns out earlier that day, a technician performing seasonal maintenance on a boiler tank on the roof had drained the tank and refilled it. But instead of directing the water out into a proper drain, he sent it down a convenient pipe that was actually a vent from our ceiling into the boiler house. The boiler was dozens of meters from my server room, but the water followed the old steel and plaster ceiling remnants over to my computers.
And this boiler water was more exciting than rain: it came with all the dissolved minerals, metals, and preservatives computers crave! I didn't lose any computers in the racks, but it killed the Liebert's control board.
If that were the case they wouldn't be saying "There is no current ETA for recovery," and "it is expected to be an extended outage. Customers are advised to failover to other regions."
You could always test this in a live environment before a region becomes open to customers.
Restoration is hard when health and safety are in question. Good luck to these ops folks <3
[1] https://www.datacenterknowledge.com/archives/2008/06/01/expl...
It was before dam (1) was built and floods were a huge problem in SPB
Our DC was intact, but the building and access was cut-off. We lost the backup diesel power generators in the flooding. Of course, grid power was cut-off.
Our DC operating team managed to shutdown all the servers and racks cleanly before UPS power was completely drained. The 4 engineers and 2 security guards then swam out of the compound in chest high waters. (I am not kidding).
When the rains subsided and the flood waters receded after a couple of days, we had to plan the restart. The facility still had to be certified by health and safety, but we needed to get the datacenter back up.
A secondary operations site that would remote-connect to the DC was brought up in 1 week since we estimated the rains to potentially continue for a few more days and cause interruptions. But the critical item for the plan to work was getting a new backup power setup. We rolled in a truck-mounted diesel generator and positioned it in the highest point in the campus (also close to our building tower that had the DC) and ran power cables to it (we had to source this and it was a challenge to do it with the time crunch and the rains).
We moved staff to other cities by bus (airport was shutdown) as part of our recovery plan, but we still needed connectivity to our DC for some of the critical processes.
Long story short, it worked.
I'll never forget the experience and the scars from this war story.
In any case, it is good you didn't have to go through a DC recovery during one of the worst disasters in the 21st century.
The question I keep asking in all DR planning sessions/table top exercises is - what would we do if we had a situation like what happened in Fukushima or in Chennai 2015. In both cases, flooding caused failure of backup power generators. Also, what do we do when we have all or partial resources, but are faced with a denial-of-premises situation (what I faced).
Warning FailedToCreateRoute 4m59s route_controller Could not create route fc61a148-b428-43fa-xxxx-xxxx 10.28.167.0/24 for node gke-xxx-xxx after 16.320065487s: googleapi: Error 503: INTERNAL_ERROR - Internal error. Please try again or contact Google Support.
Any facing something similar?Good reminder that downtime happens for many wild reasons, and you may want to take 30 seconds and set up a free website / API monitor with Heii On-Call [1] because we would have alerted you to either of these issues if they affected your app.
Really, a simple HTTP probe provides tremendous monitoring power. I already was telling people that it covered issues at the DNS, TCP, SSL certificate, load balancer, framework, and application layers. Now I will have to add “datacenter flood” as well :P
Best wishes to everyone working on europe-west-9.
[1] https://heiioncall.com/ (I recently helped build our HTTP probe background infrastructure in Crystal)
Sounds like some global control plane related to instance management operations started returning errors once one region failed. Or perhaps it was just the UI frontend?
[1] https://status.cloud.google.com/incidents/BWK7QzFBmfaZ4iztke...
I am afraid this is not true. We have nothing in europe-west-9, but problem in this region caused global problem with Cloud Console, which hit us, because we were not able to use it for several hours.
Snippert from https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf...:
"Cloud Console: Experienced a global outage, which has been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT."
europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf...
There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it has no impact on serving and they're rolling out a fix already. https://status.cloud.google.com/incidents/uSjFxRvKBheLA4Zr5q...