News for hackers. Stuff that matters.
Re:Google Cloud Incident Report – 2025-06-13 by singron on Saturday June 14, 2025 @09:09AM (#44276237)
Nearly every global outage at Google has looked vaguely like this. I.e. a bespoke system that rapidly deploys configs globally gets a bad config.

All the standard tools for binary rollouts and config pushes will typically do some kind of gradual rollout.

In some ways Google Cloud had actually greatly improved the situation since a bunch of global systems were forced to become regional and/or become much more reliable. Google also used to have short global outages that weren't publicly remarked on (at the time, if you couldn't connect to Google, you assumed your own ISP was broken), so this event wasn't as rare as you might think. Overall I don't think there is a worsening trend unless someone has a spreadsheet of incidents proving otherwise.

[I was an SRE at Google several years ago]

Re:Google Cloud Incident Report – 2025-06-13 by btown on Saturday June 14, 2025 @10:03AM (#44276507)
From the OP:

> a policy change was inserted into the regional Spanner tables that Service Control uses for policies. Given the global nature of quota management, this metadata was replicated globally within seconds

If there’s a root cause here, it’s that “given the global nature of quota management” wasn’t seen as a red flag that “quota policy changes must use the standard gradual rollout tooling.”

The baseline can’t be “the trend isn’t worsening;” the baseline should be that if global config rollouts are commonly the cause of problems, there should be increasingly elevated standards for when config systems can bypass best practices. Clearly that didn’t happen here.

Re:Google Cloud Incident Report – 2025-06-13 by palcu on Saturday June 14, 2025 @02:42PM (#44277969)
One of the problems has been that most users have requested that quotas get updated as fast as possible and that they should be consistent across regions, even for global quotas. As such people have been prioritising user experience rather than availability.

I hope the pendulum swings the other way around now in the discussion.

[disclaimer that I worked as a GCP SRE for a long time, but not left recently]

Re:Google Cloud Incident Report – 2025-06-13 by stogot on Saturday June 14, 2025 @06:47PM (#44279349)
Why did you leave? If you don’t mind asking
Re:Google Cloud Incident Report – 2025-06-13 by PNewling on Saturday June 14, 2025 @10:13PM (#44280093)
Not sure if you'll get an answer (I'd be interesting in a response as well), but from the blog in their profile it looks like they moved to be a 'member of technical staff working in the AI Reliability Engineering (AIRE) team at Anthropic'. So it might have just been an upward move to something different/more-exciting.
Re:Google Cloud Incident Report – 2025-06-13 by palcu on Sunday June 15, 2025 @08:14AM (#44281910)
I confirm. I’m still doing reliability, but in a fun and exciting way for Claude and Anthropic.