News for hackers. Stuff that matters.
Re:Google Cloud region currently down due to water intrusion by dekhn on Thursday April 27, 2023 @03:51PM (#35733635)
zones and regions have different definition in google cloud than AWS. Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.
Re:Google Cloud region currently down due to water intrusion by lamontcg on Thursday April 27, 2023 @04:25PM (#35734043)
I don't know what it is like these days but us-east AZs used to be in different datacenters that were on different flood plains and power companies. They were just on a very high capacity (for the day) fiber ring. You still had a couple miles of light-delay latency in between them. A sufficiently big enough meteor, or a powerful enough massive hurricane, could probably take out multiple ones at the same time.
Re:Google Cloud region currently down due to water intrusion by londons_explore on Thursday April 27, 2023 @04:38PM (#35734204)
I believe the Google design is one big pool of machines, perhaps spread across a few buildings, but they hope that any failure only affects a few racks.

They will arrange/move workloads such that any one customer will only see an outage in one 'zone'.

Clearly that didn't work here,

Re:Google Cloud region currently down due to water intrusion by local_crmdgeon on Thursday April 27, 2023 @05:22PM (#35734792)
That's .... that's not a great strategy. I get that it's cheap, but incidents like this will cost a LOT more with lost business.
Re:Google Cloud region currently down due to water intrusion by ericpauley on Thursday April 27, 2023 @04:19PM (#35733978)
Wow, this is a big issue! Is there any way to guarantee AWS-level physical redundancy in GCP without paying the latency/inter-region data transfer (higher than inter-zone outside US: https://cloud.google.com/vpc/network-pricing#egress-within-g...) pricing?

A nice thing about EC2 is that you're getting a pretty dumb, predictable service. There have been multi-zone or global control plane issues but the physical metal has bona fide redundancy between zones/regions.

Re:Google Cloud region currently down due to water intrusion by styren on Thursday April 27, 2023 @04:17PM (#35733964)
In the same building is quite a bit different than 10km apart, even if a huge meteor would lead to the same conclusion for both scenarios.
Re:Google Cloud region currently down due to water intrusion by deathanatos on Thursday April 27, 2023 @04:54PM (#35734417)
> A cluster represents a distinct physical infrastructure that is housed in a data center.

> Google designs zones to minimize the risk of correlated failures caused by physical infrastructure outages

And they have stated that the flood "caused a multi-cluster failure".

> Zones should be considered a single failure domain within a region.

(—GCP's documentation.)

Re:Google Cloud region currently down due to water intrusion by sitkack on Friday April 28, 2023 @09:07AM (#35740824)
"should be considered" -> we would like you to think

Zones are an artificial construct.

Re:Google Cloud region currently down due to water intrusion by illumin8 on Saturday April 29, 2023 @05:51PM (#35757601)
I guess you just discovered the difference between a GCP Zone and an AWS Availability Zone.

By definition, AWS availability zones do not share fault domains, other than a geographic region up to hundreds of miles wide. Even for services used by multiple AZs, such as transit to the Internet and other regions, there are two transit centers operating in separate fault domains.

In contrast, many GCP Zones share the same physical datacenter. What they are actually providing you are simply different racks, rows of racks, or rooms in a single physical facility. Caveat emptor.

Re:Google Cloud region currently down due to water intrusion by secondcoming on Thursday April 27, 2023 @04:57PM (#35734458)
Then why do they charge extra for cross-zonal traffic?
Re:Google Cloud region currently down due to water intrusion by kccqzy on Thursday April 27, 2023 @05:33PM (#35734902)
Presumably for inter-zonal traffic, it uses up bandwidth on their B4 network, but intra-zonal traffic does not and has basically unlimited bandwidth.

B4: https://storage.googleapis.com/pub-tools-public-publication-...

Re:Google Cloud region currently down due to water intrusion by dilyevsky on Thursday April 27, 2023 @05:42PM (#35734996)
This is not going through b4 but inter-zonal links are still choke points because they are separate networks