Re:What Flips Your Bit: Cosmic Ray Errors at Mozilla
by dijit on Thursday April 14, 2022 @03:55AM
(#31024375)
In 2014 I worked on a video game that loaded 1,000 instances of a mutable game world into memory, each machine had 256G of ram (about 210G was allocated to the game server) and I ran about 2,500~ of those.
The large mutable state like this tends to stress the memory bandwidth, so I’m not sure it’s related, but we used ECC memory and saw a seriously insane number of corrected bit flips.
Somewhere in the order of 100/s over the whole fleet as a baseline.
there were times that this rate spiked due to a faulty RAM stick, but if you look at the distribution it was fairly even, though there were fairly random hotspots.
After observing this I decided I didn’t want to live without ECC on my personal life, to my surprise there is no laptop on the market that supports ECC. Or at least none that I could find.
This issue goes away with DDR5 and built-in ECC though, or, our visibility of it does. So, I’m happy.
The large mutable state like this tends to stress the memory bandwidth, so I’m not sure it’s related, but we used ECC memory and saw a seriously insane number of corrected bit flips.
Somewhere in the order of 100/s over the whole fleet as a baseline.
there were times that this rate spiked due to a faulty RAM stick, but if you look at the distribution it was fairly even, though there were fairly random hotspots.
After observing this I decided I didn’t want to live without ECC on my personal life, to my surprise there is no laptop on the market that supports ECC. Or at least none that I could find.
This issue goes away with DDR5 and built-in ECC though, or, our visibility of it does. So, I’m happy.