News for hackers. Stuff that matters.
Re:Google Cloud Incident Report – 2025-06-13 by blibble on Saturday June 14, 2025 @08:05AM (#44275870)
this is really amateur level stuff: NPEs, no error handling, no exponential backoff, no test coverage, no testing in staging, no gradual rollout, fail deadly

I read their SRE books, all of this stuff is in there: https://sre.google/sre-book/table-of-contents/ https://google.github.io/building-secure-and-reliable-system...

have standards slipped? or was the book just marketing

Re:Google Cloud Incident Report – 2025-06-13 by marcinzm on Saturday June 14, 2025 @08:12AM (#44275917)
As an outsider my quick guess is that at some point after enough layoffs and the CEO accusing everyone of being lazy, people focus on speed/perceived output over quality. After a while the culture shifts so if you block such things then you're the problem and will be ostracized.
Re:Google Cloud Incident Report – 2025-06-13 by gjsman-1000 on Saturday June 14, 2025 @09:08AM (#44276231)
As an outsider, what I perceive is quite different:

HN likes to pretend that FAANG is the pinnacle of existence. The best engineers, the best standards, the most “that wouldn’t have happened here,” the yardstick by which all companies should be measured for engineering prowess.

Incidents like this repeatedly happening reveal that’s mostly a myth. They aren’t much smarter, their standards are somewhat wishful thinking, their accomplishments are mostly rooted in the problems they needed to solve just like any other company.

Re:Google Cloud Incident Report – 2025-06-13 by spacemadness on Saturday June 14, 2025 @12:07PM (#44277182)
That’s just PR that serves these companies. I’ve never seen them that way. The stupid avoidable bugs and terrible UX in a lot of their products tells you enough at the surface level. What’s true is these companies do hire some amazing specialists but that doesn’t make them the pinnacle of engineering overall.
Re:Google Cloud Incident Report – 2025-06-13 by dvfjsdhgfv on Saturday June 14, 2025 @11:16AM (#44276891)
You might be right to some extend, but not entirely. For example, there have been almost no incidents in AWS where one customer would be able to access the data of another customer because of AWS fault. The cases so far like Superglue etc. were very limited and IMHO AWS security is quite solid.

So I would say there is a difference between AWS architects and engineers (although I know first hand that certain things are subobtimal, but...) and those of several other companies who have less customers but experienced successful attacks (or data loss). Even if you take Microsoft, there is huge difference in security posture between AWS and Azure (and I say this as a big fan of the so-called "private cloud" (previously know as just your own infra)).

Re:Google Cloud Incident Report – 2025-06-13 by belter on Saturday June 14, 2025 @02:29PM (#44277889)
Every time I hear a company uses Azure I check public leak sites for their data...

https://www.lastweekinaws.com/blog/azures_vulnerabilities_ar...

Re:Google Cloud Incident Report – 2025-06-13 by reaperducer on Saturday June 14, 2025 @03:46PM (#44278374)
Every time I hear a company uses Azure I check public leak sites for their data...

https://www.lastweekinaws.com/blog/azures_vulnerabilities_ar...

(2022)

Re:Google Cloud Incident Report – 2025-06-13 by belter on Saturday June 14, 2025 @06:26PM (#44279251)
2025... -> "Critical 10/10 Microsoft Cloud Security Vulnerability Confirmed" - https://www.forbes.com/sites/daveywinder/2025/05/11/microsof...
Re:Google Cloud Incident Report – 2025-06-13 by motorest on Saturday June 14, 2025 @11:30AM (#44276968)
> Incidents like this repeatedly happening reveal that’s mostly a myth. They aren’t much smarter, their standards are somewhat wishful thinking, their accomplishments are mostly rooted in the problems they needed to solve just like any other company.

I think you're only seeing what you want to see, because somehow bringing FANG engineers down a peg makes you feel better?

A broken deployment due to a once-in-a-lifetime configuration change in a project that wasn't allocated engineering effort to allow more robust and resilient deployment modes doesn't turn any engineer into an incompetent fool. Sometimes you need to flip a switch, and you can't spare a team working one year to refactor the whole thing.

Re:Google Cloud Incident Report – 2025-06-13 by floydnoel on Saturday June 14, 2025 @02:03PM (#44277775)
> Sometimes you need to flip a switch, and you can't spare a team working one year to refactor the whole thing.

This seems to imply that the person in charge at G was right to cause this outage... and that Google is very short-staffed and too poor to afford to do proper engineering work?

Somehow that doesn't inspire confidence in their engineering prowess. Sure seems to me that bad engineering leadership decisions is equivalent to bad engineering.

Re:Google Cloud Incident Report – 2025-06-13 by davidcbc on Sunday June 15, 2025 @12:22PM (#44283139)
> Google is very short-staffed

This has been the case for many teams since January 2023

Re:Google Cloud Incident Report – 2025-06-13 by flaminHotSpeedo on Sunday June 15, 2025 @02:08PM (#44283927)
> Sometimes you need to flip a switch

If anyone needs to flip a global switch, and can't convince their leadership to allocate the resources to do it safely, engineering culture is dead, at least locally to that system.

But I'm not convinced lack of headcount was the problem here, the incident report makes it sound like there's an established pattern for feature flagging even for global changes like this.

Putting aside the fact that this team seems unperturbed by global deployments and all the other scary things, high impact changes should use every mechanism available to shrink fault containers. It would be inexcusable to roll this change out without tha feature flag mechanism if this were a regional roll out.

Skipping the feature flag when this is global is simply incomprehensible. It goes beyond headcount, it should never have been considered in the first place.

Re:Google Cloud Incident Report – 2025-06-13 by tough on Saturday June 14, 2025 @08:06AM (#44275879)
someone vibe coded a push to prod on friday?
Re:Google Cloud Incident Report – 2025-06-13 by koakuma-chan on Saturday June 14, 2025 @08:29AM (#44276022)
Ironically Gemini wouldn’t forget a null check
Re:Google Cloud Incident Report – 2025-06-13 by maigret on Saturday June 14, 2025 @08:46AM (#44276109)
Try to a few Lighthouse measurements on their web pages and you’ll see they don’t maintain the highest engineering standards.
Re:Google Cloud Incident Report – 2025-06-13 by koakuma-chan on Saturday June 14, 2025 @08:49AM (#44276126)
Yep, people (including me) like to shit on Next.js, but I have a fairly complex app, and it’s still at 100 100 100 100
Re:Google Cloud Incident Report – 2025-06-13 by kyrra on Saturday June 14, 2025 @08:50AM (#44276128)
I wish they would share more details here. Your take isn't fully correct. There was testing, just not for the bad input (the blank fields in the policy). They also didn't say there was no testing in staging, just that a flag would have caught it.

Opinions are my own.

Re:Google Cloud Incident Report – 2025-06-13 by fidotron on Saturday June 14, 2025 @08:54AM (#44276153)
> There was testing, just not for the bad input (the blank fields in the policy).

But if you change a schema, be it DB, protobuf, whatever, this is the major thing your tests should be covering.

This is why people are so amazed by it.

Re:Google Cloud Incident Report – 2025-06-13 by kyrra on Saturday June 14, 2025 @08:58AM (#44276184)
Sorry, I meant that was unit testing.

The document also doesn't say there wasn't testing in staging or prod.

Re:Google Cloud Incident Report – 2025-06-13 by fidotron on Saturday June 14, 2025 @09:04AM (#44276209)
> Sorry, I meant that was unit testing.

So do we. If you change the schema it doesn't matter if it's unit testing, you either have a validation stage for all data coming in beforehand to check these assumptions (the way many python people use pydantic on incoming data, for example), and that must be updated for every change, or every unit must be defensive against this sort of thing.

Re:Google Cloud Incident Report – 2025-06-13 by ajb on Saturday June 14, 2025 @08:56AM (#44276171)
...the constant familiarity with even the most dangerous instruments soon makes men loose their first caution in handling them; they readily, therefore, come to think that the rules laid down for their guidance are unnecessarily strict - report on the explosion of a gunpowder magazine at Erith, 1864
Re:Google Cloud Incident Report – 2025-06-13 by belter on Saturday June 14, 2025 @09:12AM (#44276257)
The incredible reliability and high standards in most of the air travel industry proves this wrong.
Re:Google Cloud Incident Report – 2025-06-13 by ajb on Saturday June 14, 2025 @11:03AM (#44276826)
Yes, I don't say that a quote from 1864 is the last word on work culture. Nevetheless, it does capture something of human nature: what is now known as "normalisation of deviance".
Re:Google Cloud Incident Report – 2025-06-13 by mplanchard on Saturday June 14, 2025 @11:17AM (#44276896)
I guess you’ve never heard pilots complain about the FAA
Re:Google Cloud Incident Report – 2025-06-13 by gjsman-1000 on Saturday June 14, 2025 @09:01AM (#44276199)
Standards fallen?

Google’s standards, and from what I can tell, most FAANG standards are like beauty filters on Instagram. Judging yourself, or any company, against them is delusional.

Re:Google Cloud Incident Report – 2025-06-13 by belter on Saturday June 14, 2025 @09:09AM (#44276235)
- You do know their AZs are just firewalls across the same datacenter?

- And they used machines without ECC and their index got corrupted because of it? And instead of hiding the head in shame and getting lessons from IBM old timers they published a paper about it?

- What really accelerated the demise of Google+ was that an API issue allowed the harvesting of private profile fields for millions of users, and they hid that for months fearing the backlash...

Dont worry, you will have plenty more outages from the land of we only hire the best....

Re:Google Cloud Incident Report – 2025-06-13 by atombender on Saturday June 14, 2025 @03:46PM (#44278377)
I wonder if machines without ECC could perhaps explain why our apps periodically see TCP streams with scrambled contents.

On GKE, we see different services (like Postgres and NATS) running on the same VM in different containers receive/send stream contents (e.g. HTTP responses) where the packets of the stream have been mangled with the contents of other packets. We've been seeing it since 2024, and all the investigation we've done points to something outside our apps and deeper in the system. We've only seen it in one Kubernetes cluster, and it lasts 2-3 hours and then magically resolves itself; draining the node also fixes it.

If there are physical nodes with faulty RAM, I bet something like this could happen. Or there's a bug in their SDN or their patched version of the Linux kernel.

Re:Google Cloud Incident Report – 2025-06-13 by belter on Saturday June 14, 2025 @06:24PM (#44279243)
You could start by running similar parallel infra on AWS that is ECC everywhere...And also check the corrupted TCP streams for single-bit flip patterns, also maybe correlate timing with memory pressure as that would be what and when the RAM errors would typically show. If its more than just bit flips, could be something else.
Re:Google Cloud Incident Report – 2025-06-13 by atombender on Saturday June 14, 2025 @06:32PM (#44279278)
It's a lot more, we basically see random fragments of other parallel TCP streams.
Re:Google Cloud Incident Report – 2025-06-13 by belter on Sunday June 15, 2025 @09:58AM (#44282326)
Sounds like a bug on their custom CNI...or Kernel. I would implement checksum validation in the application layer as a troubleshooting measure and try to correlate the errors to other events. Is that during their maintenance windows?
Re:Google Cloud Incident Report – 2025-06-13 by atombender on Sunday June 15, 2025 @10:29AM (#44282463)
I've checked against VM migration events and they don't match. I've not actually investigated whether there is any other type of maintenance that could affect nodes. Cluster upgrades would be one, I suppose.

Hm, checksum validation isn't really possible since these are protocols out of our control, like Postgres, NATS, and various other HTTP-based services.

But we could write some kind of simple canary process constantly doing dummy network access and detecting corruption.

Re:Google Cloud Incident Report – 2025-06-13 by mnordhoff on Sunday June 15, 2025 @12:07PM (#44283038)
This sounds like a very serious security vulnerability...?
Re:Google Cloud Incident Report – 2025-06-13 by singron on Saturday June 14, 2025 @09:09AM (#44276237)
Nearly every global outage at Google has looked vaguely like this. I.e. a bespoke system that rapidly deploys configs globally gets a bad config.

All the standard tools for binary rollouts and config pushes will typically do some kind of gradual rollout.

In some ways Google Cloud had actually greatly improved the situation since a bunch of global systems were forced to become regional and/or become much more reliable. Google also used to have short global outages that weren't publicly remarked on (at the time, if you couldn't connect to Google, you assumed your own ISP was broken), so this event wasn't as rare as you might think. Overall I don't think there is a worsening trend unless someone has a spreadsheet of incidents proving otherwise.

[I was an SRE at Google several years ago]

Re:Google Cloud Incident Report – 2025-06-13 by btown on Saturday June 14, 2025 @10:03AM (#44276507)
From the OP:

> a policy change was inserted into the regional Spanner tables that Service Control uses for policies. Given the global nature of quota management, this metadata was replicated globally within seconds

If there’s a root cause here, it’s that “given the global nature of quota management” wasn’t seen as a red flag that “quota policy changes must use the standard gradual rollout tooling.”

The baseline can’t be “the trend isn’t worsening;” the baseline should be that if global config rollouts are commonly the cause of problems, there should be increasingly elevated standards for when config systems can bypass best practices. Clearly that didn’t happen here.

Re:Google Cloud Incident Report – 2025-06-13 by palcu on Saturday June 14, 2025 @02:42PM (#44277969)
One of the problems has been that most users have requested that quotas get updated as fast as possible and that they should be consistent across regions, even for global quotas. As such people have been prioritising user experience rather than availability.

I hope the pendulum swings the other way around now in the discussion.

[disclaimer that I worked as a GCP SRE for a long time, but not left recently]

Re:Google Cloud Incident Report – 2025-06-13 by stogot on Saturday June 14, 2025 @06:47PM (#44279349)
Why did you leave? If you don’t mind asking
Re:Google Cloud Incident Report – 2025-06-13 by PNewling on Saturday June 14, 2025 @10:13PM (#44280093)
Not sure if you'll get an answer (I'd be interesting in a response as well), but from the blog in their profile it looks like they moved to be a 'member of technical staff working in the AI Reliability Engineering (AIRE) team at Anthropic'. So it might have just been an upward move to something different/more-exciting.
Re:Google Cloud Incident Report – 2025-06-13 by palcu on Sunday June 15, 2025 @08:14AM (#44281910)
I confirm. I’m still doing reliability, but in a fun and exciting way for Claude and Anthropic.
Re:Google Cloud Incident Report – 2025-06-13 by perryizgr8 on Saturday June 14, 2025 @10:24AM (#44276632)
At google scale, if their standards were not sky high, such incidents would be happening daily. That it happens once in a blue moon indicates that they are really meticulous with all those processes and safeguards almost all the time.
Re:Google Cloud Incident Report – 2025-06-13 by gyesxnuibh on Saturday June 14, 2025 @10:27AM (#44276651)
That book was written with 40% of the engineers compared to when I left a couple years ago (not sure how many now with the layoffs now). I'm guessing those hires haven't read it yet. So yeah, reads like standards slipping to me.
Re:Google Cloud Incident Report – 2025-06-13 by rincebrain on Sunday June 15, 2025 @06:54AM (#44281636)
IMO, it's that any defense, humans or automated, is imperfect, and life is a series of tradeoffs.

You can write as many unit tests as you want, and integration tests that your system works as you expect on sample data, static analysis to scream if you're doing something visibly unsafe, staged rollout from nightly builds to production, and so on and so on, but eventually, at large enough scale, you're going to find a gap in those layered safety measures, and if you're unlucky, it's going to be a gap in all of them at once.

It's the same reasoning from said book as why getting another nine is always going to involve much more work than the previous ones - eventually you're doing things like setting up complete copies of your stack running stable builds from months ago and replaying all the traffic to them in order to be able to fail over to them on a moment's notice, meaning that you also can't roll out new features until the backup copies support it too, and that's a level of cost/benefit that nobody can pay if the service is large enough.

When working on OpenZFS, a number of bugs have come from things like "this code in isolation works as expected, but an edge case we didn't know about in data written 10 years ago came up", or "this range of Red Hat kernels from 3 Red Hat releases ago has buggy behavior, and since we test on the latest kernel of that release, we didn't catch it".

Eventually, if there's enough complexity in the system, you cannot feasibly test even all the variation you know about, so you make tradeoffs based on what gets enough benefit for the cost.

(I'm an SRE at Google, not on any team related to this incident, all opinions unofficial/my own, etc.)