Re:Google Cloud Incident Report – 2025-06-13
by atombender on Saturday June 14, 2025 @06:32PM
(#44279278)
It's a lot more, we basically see random fragments of other parallel TCP streams.
[ Parent
| on HN ]
Sounds like a bug on their custom CNI...or Kernel. I would implement checksum validation in the application layer as a troubleshooting measure and try to correlate the errors to other events. Is that during their maintenance windows?
Re:Google Cloud Incident Report – 2025-06-13
by atombender on Sunday June 15, 2025 @10:29AM
(#44282463)
I've checked against VM migration events and they don't match. I've not actually investigated whether there is any other type of maintenance that could affect nodes. Cluster upgrades would be one, I suppose.
Hm, checksum validation isn't really possible since these are protocols out of our control, like Postgres, NATS, and various other HTTP-based services.
But we could write some kind of simple canary process constantly doing dummy network access and detecting corruption.