News for hackers. Stuff that matters.
Re:Google Cloud Incident Report – 2025-06-13 by atombender on Saturday June 14, 2025 @06:32PM (#44279278)
It's a lot more, we basically see random fragments of other parallel TCP streams.
Re:Google Cloud Incident Report – 2025-06-13 by belter on Sunday June 15, 2025 @09:58AM (#44282326)
Sounds like a bug on their custom CNI...or Kernel. I would implement checksum validation in the application layer as a troubleshooting measure and try to correlate the errors to other events. Is that during their maintenance windows?
Re:Google Cloud Incident Report – 2025-06-13 by atombender on Sunday June 15, 2025 @10:29AM (#44282463)
I've checked against VM migration events and they don't match. I've not actually investigated whether there is any other type of maintenance that could affect nodes. Cluster upgrades would be one, I suppose.

Hm, checksum validation isn't really possible since these are protocols out of our control, like Postgres, NATS, and various other HTTP-based services.

But we could write some kind of simple canary process constantly doing dummy network access and detecting corruption.