A server with 64 GB of ECC ram sitting at an altitude of 3.2 km on the Greenland ice sheet is reporting... 0 bit errors (whether correctable or uncorrectable) in the 244 days it's been up.

A server with 16 GB of ECC ram at an altitude of 3.8 km in California is reporting.... 0 bit errors in the 146 days it's been up.

Maybe I shouldn't believe what /sys/devices/system/edac/mc is reporting? These are EL8 systems...

Bryan Cantrill mentions in this talk that they saw no correctable errors, until they suddenly got uncorrectable errors. Turns out that the firmware was hiding the fact that it had been correcting ECC errors the whole time. https://youtu.be/vvZA9n3e5pc?t=1193

I've always been a bit skeptical of published numbers. I usually just chalk it up to vastly different operating conditions and scale.

On my home server w/ ECC you can check the corrected and uncorrected (multibit) errors. Assuming my Ryzen is correctly reporting them to Linux, I have 0 errors corrected and 0 uncorrected with a 80 day uptime. I've checked a few other times and never seen an error. Others with ECC often report the same.

My understanding of modern RAM is that it has checks built in to the modules which are somewhat equivalent to ECC already (the correcting part, not the reporting part). Which is a necessity in order to hit the density we are at today.

DDR5 does and so does the raspberry pi memory.

But DDR4 and before does not I think.

>I therefore doubt that the estimation of "1/256MB/month" is correct

As someone who did incredibly poorly in high school physics, this line in the article bothered me as well: the study is from the 1990s when the density of memory would have been much lower. I would think the percentage per megabyte has dropped significantly in 30 or so years. It also assumes a constant form factor for the memory, doesn't it?

The energy needed to flip a bit has drastically lowered since the 1990s, and increased density makes bit flip rates increase at some point since a single physical event can flip multiple bits.

So it's not so simple.

Thanks. So I was sort of right only in the exact opposite direction!

> I therefore doubt that the estimation of "1/256MB/month" is correct

The probability is related to the physical volume the memory takes, since it's caused by a physical particle going through that volume. So, this rate will continuously drop as memory density increases.

My understanding is that newer/denser transistors are actually more susceptible to single event upsets, because it takes less energy to flip a bit.

The last time I did research into this, this was the most concrete real world example I could find, which works out to ~41 bitflips/GiB-month:

> Jaguar had 360 terabytes of main memory [and] was logging ECC errors at a rate of 350 per minute

Source: https://spectrum.ieee.org/how-to-kill-a-supercomputer-dirty-...

This is throwing up a soft paywall for me FWIW - I get up to "...couldn’t run more than an hour or so without crashing." and then the article text stops, there's a "become a member" interstitial and then I'm fed with tons of headlines for different articles instead.

Because of the sharp contrast with the interstitial block I thought it was like a footer, and then a skippable/ignorable part of the layout, and spent some time confusedly scrolling up and down thinking the content was past it, then I realized the "Keep Reading" bits were links to other articles... wow what a mess.

</rant>

In 2014 I worked on a video game that loaded 1,000 instances of a mutable game world into memory, each machine had 256G of ram (about 210G was allocated to the game server) and I ran about 2,500~ of those.

The large mutable state like this tends to stress the memory bandwidth, so I’m not sure it’s related, but we used ECC memory and saw a seriously insane number of corrected bit flips.

Somewhere in the order of 100/s over the whole fleet as a baseline.

there were times that this rate spiked due to a faulty RAM stick, but if you look at the distribution it was fairly even, though there were fairly random hotspots.

After observing this I decided I didn’t want to live without ECC on my personal life, to my surprise there is no laptop on the market that supports ECC. Or at least none that I could find.

This issue goes away with DDR5 and built-in ECC though, or, our visibility of it does. So, I’m happy.

Agree. I have once learn subject of cosmic rays, and found estimation, that every square meter suffer HUNDREDS events per second, of course mostly weak events.

What I really see - on digital photo dark current test videos, usually could see few flashes.

So looks like, modern ram has some sort of ecc inside chip.

And yes, digital photo sensors much better suited for detecting of cosmic rays, even simplest, sure much better (semi)professional, which could tuned to make long exposition. But even on book webcam, or on smart front camera, could see events when low light (or if close lens with some opaque material).