News for hackers. Stuff that matters.
Re:Post Mortem of Google Outage on 14 December 2020 by felixhuttmann on Friday December 18, 2020 @09:38PM (#25474489)
I often hear 'aim for elimination of global dependencies', but the reality is that there is no way around global dependencies. AWS STS or IAM is just as global as google's. The difference is that google more often builds with some form of guaranteed read-after-write consistency, while AWS is more often 'fail open'. For example, if you remove a permission from a user in GCP, you are guaranteed consistency within 7 minutes [1], while with AWS IAM, your permissions may be arbitrarily stale. This means that when the GCP IAM database leader fails, all operations will globally fail after 7 minutes, while with AWS IAM, everything continues to work when the leader fails, but as an AWS customer, you can never be sure that some policy change has actually become effective.

In general, AWS more often shifts the harder parts of global distributed systems onto their customers, rather than solving them for their customers, like GCP does. For example, GCP cloud storage (s3 equivalent) and datastore (nosql database) provide strongly consistent operations in multi-region configurations, while dynamodb and s3 have only eventually consistent replication across regions; and google's VPCs, message queues, console VM listings, and loadbalancers are global, while AWS's are regional.

[1] https://cloud.google.com/iam/docs/faq#access_revoke

Re:Post Mortem of Google Outage on 14 December 2020 by inopinatus on Friday December 18, 2020 @11:29PM (#25475038)
> In general, AWS more often shifts the harder parts of global distributed systems onto their customers, rather than solving them for their customers, like GCP does.

Choice of language in representing this is rather telling, because AWS can (and does) pitch this as a strength, viz. that regionalisation helps customers (especially, significantly, bigco enterprise customers) reason about the possible failure modes, and thereby contain the blast radius of component failure.

They'd never comment on competitors in public, but the clear implication is that apparently global services merely gloss over the risks, they don't resolve them, and eventually it'll blow up in your face, or someone's face at least.

> there is no way around global dependencies

This sounds more like a challenge than an assertion. In my very long experience of tech, anyone who ever said, "you can't do that", eventually ate their hat.

Re:Post Mortem of Google Outage on 14 December 2020 by erhk on Saturday December 19, 2020 @08:39AM (#25477314)
Slow rollouts are a security hole.
Re:Post Mortem of Google Outage on 14 December 2020 by cle on Friday December 18, 2020 @10:28PM (#25474728)
Side note: AWS STS has had regional endpoints for years. The global endpoint is vestigial at this point. I didn't glean anything special about Google's endpoint that requires it to be globalized like this, but I can't really criticize it without knowing the details.
Re:Post Mortem of Google Outage on 14 December 2020 by silentsea90 on Friday December 18, 2020 @09:58PM (#25474570)
S3 is strongly consistent. https://aws.amazon.com/s3/consistency/

Which of Google's nosql db provides strong consistency - bigtable? Just confirming

Re:Post Mortem of Google Outage on 14 December 2020 by felixhuttmann on Friday December 18, 2020 @10:08PM (#25474622)
S3 is newly strongly consistent within a single region since last reinvent or so (google cloud storage has been strongly consistent for much longer). However, the cross-region replication for s3 is based on 'copying' [1] so presumably async and not strongly consistent.

GCP datastore and firestore are strongly consistent nosql databases that are available in multi-region configurations [2].

[1] https://docs.aws.amazon.com/AmazonS3/latest/dev/replication.... [2] https://cloud.google.com/datastore/docs/locations

Re:Post Mortem of Google Outage on 14 December 2020 by frankchn on Saturday December 19, 2020 @12:54AM (#25475433)
S3 became strongly consistent only recently (https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...) while I think GCS and Azure Blob Storage has strong read-after-write consistency for a while now.

In any case, Cloud Spanner provides strong consistency in multi-region deployments.

Re:Post Mortem of Google Outage on 14 December 2020 by dijit on Saturday December 19, 2020 @07:13AM (#25476946)
And GCP storage buckets has been built on top of spanner sine 2018- giving the same guarantees.

If anything. AWS is playing catch up.

Re:Post Mortem of Google Outage on 14 December 2020 by Terretta on Saturday December 19, 2020 @09:34AM (#25477622)
AWS’s regionality and stronger region separation boundaries are a huge selling point for regulated (data regulation) industries and enterprises.

A bank, for instance, may be required to prove it cannot replicate customer data across regions, and that no third party provider will replicate its data using their own BCM or DR systems.

Regardless of CSP, startups should think about rules on movement of data among data jurisdictions (such as GDPR) and architect accordingly.

Re:Post Mortem of Google Outage on 14 December 2020 by baskire on Sunday December 20, 2020 @10:29AM (#25486877)
S3 has strong consistency of list operations now.