News for hackers. Stuff that matters.
Re:Google Cloud region currently down due to water intrusion by t0mas88 on Thursday April 27, 2023 @03:44PM (#35733544)
I can't ignore the feeling that Google Cloud is sub par compared to AWS. How did this again cause a multi zone failure. Why haven't they fixed those dependencies the last few times they had a full region failure.
Re:Google Cloud region currently down due to water intrusion by ec109685 on Thursday April 27, 2023 @04:40PM (#35734224)
It was not only a region outage, but a global outage!

"GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating... "

Embarrassing.

Re:Google Cloud region currently down due to water intrusion by dekhn on Thursday April 27, 2023 @03:51PM (#35733635)
zones and regions have different definition in google cloud than AWS. Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.
Re:Google Cloud region currently down due to water intrusion by lamontcg on Thursday April 27, 2023 @04:25PM (#35734043)
I don't know what it is like these days but us-east AZs used to be in different datacenters that were on different flood plains and power companies. They were just on a very high capacity (for the day) fiber ring. You still had a couple miles of light-delay latency in between them. A sufficiently big enough meteor, or a powerful enough massive hurricane, could probably take out multiple ones at the same time.
Re:Google Cloud region currently down due to water intrusion by londons_explore on Thursday April 27, 2023 @04:38PM (#35734204)
I believe the Google design is one big pool of machines, perhaps spread across a few buildings, but they hope that any failure only affects a few racks.

They will arrange/move workloads such that any one customer will only see an outage in one 'zone'.

Clearly that didn't work here,

Re:Google Cloud region currently down due to water intrusion by local_crmdgeon on Thursday April 27, 2023 @05:22PM (#35734792)
That's .... that's not a great strategy. I get that it's cheap, but incidents like this will cost a LOT more with lost business.
Re:Google Cloud region currently down due to water intrusion by ericpauley on Thursday April 27, 2023 @04:19PM (#35733978)
Wow, this is a big issue! Is there any way to guarantee AWS-level physical redundancy in GCP without paying the latency/inter-region data transfer (higher than inter-zone outside US: https://cloud.google.com/vpc/network-pricing#egress-within-g...) pricing?

A nice thing about EC2 is that you're getting a pretty dumb, predictable service. There have been multi-zone or global control plane issues but the physical metal has bona fide redundancy between zones/regions.

Re:Google Cloud region currently down due to water intrusion by styren on Thursday April 27, 2023 @04:17PM (#35733964)
In the same building is quite a bit different than 10km apart, even if a huge meteor would lead to the same conclusion for both scenarios.
Re:Google Cloud region currently down due to water intrusion by deathanatos on Thursday April 27, 2023 @04:54PM (#35734417)
> A cluster represents a distinct physical infrastructure that is housed in a data center.

> Google designs zones to minimize the risk of correlated failures caused by physical infrastructure outages

And they have stated that the flood "caused a multi-cluster failure".

> Zones should be considered a single failure domain within a region.

(—GCP's documentation.)

Re:Google Cloud region currently down due to water intrusion by sitkack on Friday April 28, 2023 @09:07AM (#35740824)
"should be considered" -> we would like you to think

Zones are an artificial construct.

Re:Google Cloud region currently down due to water intrusion by illumin8 on Saturday April 29, 2023 @05:51PM (#35757601)
I guess you just discovered the difference between a GCP Zone and an AWS Availability Zone.

By definition, AWS availability zones do not share fault domains, other than a geographic region up to hundreds of miles wide. Even for services used by multiple AZs, such as transit to the Internet and other regions, there are two transit centers operating in separate fault domains.

In contrast, many GCP Zones share the same physical datacenter. What they are actually providing you are simply different racks, rows of racks, or rooms in a single physical facility. Caveat emptor.

Re:Google Cloud region currently down due to water intrusion by secondcoming on Thursday April 27, 2023 @04:57PM (#35734458)
Then why do they charge extra for cross-zonal traffic?
Re:Google Cloud region currently down due to water intrusion by kccqzy on Thursday April 27, 2023 @05:33PM (#35734902)
Presumably for inter-zonal traffic, it uses up bandwidth on their B4 network, but intra-zonal traffic does not and has basically unlimited bandwidth.

B4: https://storage.googleapis.com/pub-tools-public-publication-...

Re:Google Cloud region currently down due to water intrusion by dilyevsky on Thursday April 27, 2023 @05:42PM (#35734996)
This is not going through b4 but inter-zonal links are still choke points because they are separate networks
Re:Google Cloud region currently down due to water intrusion by influx on Thursday April 27, 2023 @04:48PM (#35734346)
AWS has had datacenters that were flooded, but they failed over and physically moved racks via trucks. Customers never knew.
Re:Google Cloud region currently down due to water intrusion by local_crmdgeon on Thursday April 27, 2023 @05:27PM (#35734848)
This has happened much more than people know.
Re:Google Cloud region currently down due to water intrusion by Aeolun on Thursday April 27, 2023 @08:32PM (#35736212)
If it happens without me knowing then they’re doing their job very well.
Re:Google Cloud region currently down due to water intrusion by nonethewiser on Thursday April 27, 2023 @04:55PM (#35734432)
> I can't ignore the feeling that Google Cloud is sub par compared to AWS.

That goes without saying at this point. More importantly, it’s proven worse than Azure.

Re:Google Cloud region currently down due to water intrusion by outworlder on Thursday April 27, 2023 @06:25PM (#35735316)
Depending on the metric. I have not seen any incidents where the isolation between customers was breach. Azure had several. Their compute offerings are better. We could go on.

On the other hand, Azure was(and still is) upfront about not having AZs - now that they have rolled out, hopefully those are not in the same building.

Re:Google Cloud region currently down due to water intrusion by 1letterunixname on Friday April 28, 2023 @01:13AM (#35737669)
Well, GCP did fire most of their senior customer techs 18 months ago.

Alphabet is most ruled by arbitrary fiat of empire-builders.

Killing products and footgun changes are standard.