Re:Binaries
by kridsdale1 on Monday December 29, 2025 @09:44PM
(#46428933)
As a developer depending on the infrastructure and systems you guys make reliable every day inside Google, Bless You. Truly.
When Forge has a problem, I might as well go on a nature hike.
It's pretty challenging for software to defend against SEUs corrupting memory, especially when retrofitting an existing design like Linux. While operating Forge, we saw plenty of machines miscompute stuff, and we definitely worried about garbage getting into our caches. But my recollection is that the main cause was individual bad CPUs. We would reuse files in tmpfs for days without reverifying their checksums, and while we considered adding a scrubber, we never saw evidence that it would have caught much.
Maybe the CPU failures were actually due to radiation damage, but as they tended to be fairly sticky, my guess is something more like electromigration.