News for hackers. Stuff that matters.

A comment on Cooling related failure (in Google London DC).

Re:Cooling related failure (in Google London DC) by dozzman on Wednesday July 20, 2022 @09:06AM (#32165224)
My main confusion with this downtime is that neither their Cloud SQL nor Redis offerings managed to complete fail over despite my org having high availability enabled on both of those plans. Is there something I'm missing here? I would've suspected that failover would kick in for high availability instances and cause minimal downtime however its been almost 24 hours and our Cloud SQL instance is still stuck on attempting to fail over, not to mention that it comes at a premium. Wondering if anyone can help me understand what I'm missing or if the failover behaviour is not working. We've made our own workarounds in the mean time.

Relevant docs I've checked for behaviour:

https://cloud.google.com/memorystore/docs/redis/high-availab...

https://cloud.google.com/sql/docs/mysql/high-availability

EDIT: Have found out from our ops team that the SQL instance recovered around 3am so it was down for approximately 9 hours -- which is still totally useless for something deemed HA.

[ Parent | on HN ]
Re:Cooling related failure (in Google London DC) by Tostino on Wednesday July 20, 2022 @11:13AM (#32166919)
Seems like they need to start issuing some refunds/credits.
[ Parent | on HN ]
Re:Cooling related failure (in Google London DC) by cshou on Wednesday July 20, 2022 @10:08AM (#32166003)
IIUC, HA setting only failover across *zones in the same region*. If the whole region is down, HA won’t be helpful. In this case, the London data center is the region.
[ Parent | on HN ]
Re:Cooling related failure (in Google London DC) by giggsey on Wednesday July 20, 2022 @10:53AM (#32166588)
The region wasn't down though. Only one zone was down?

From earlier in the incident history:

> Cloud SQL:

> Impact/Diagnosis: Non-HA instances backed by europe-west2-a are hard-down in europe-west2-a. HA instances that were in europe-west2-a when the incident started, are down with stuck failovers.

[ Parent | on HN ]
Re:Cooling related failure (in Google London DC) by nogbit on Wednesday July 20, 2022 @10:31AM (#32166280)
That’s expected, Cloud SQL is not multi region. Clouds define HA as being multizonal, which you were.

Try Spanner if one region is not enough.

[ Parent | on HN ]
Re:Cooling related failure (in Google London DC) by dozzman on Wednesday July 20, 2022 @11:04AM (#32166780)
The whole region wasn’t down though, only zone europe-west2-a so AFAICT high availability should’ve covered this particular instance of outage.
[ Parent | on HN ]
Re:Cooling related failure (in Google London DC) by onphonenow on Wednesday July 20, 2022 @09:37AM (#32165613)
That’s pretty terrible!
[ Parent | on HN ]