News for hackers. Stuff that matters.
Re:ECC matters by cbanek on Sunday January 03, 2021 @11:47AM (#25622851)
As someone who has had to read thousands of random game crash reports from all over the interwebs (you know when Windows says you might want to send that crash log? like that), I totally agree.

Of all the things to be worried about, like OS bugs, bad hardware configuration, etc. bad memory is one of those really troubling things. You look at the code and say "it's can't make it here, because this was set" but when you can't trust your memory you can't trust anything.

And as the timeline goes to infinity, you may also get one of these reports and be asked to fix it... good luck.

Re:ECC matters by TonyTrapp on Sunday January 03, 2021 @07:17PM (#25626613)
One of my favourite stories about this type if stuff is from Patrick Wyatt on developing Guild Wars: https://www.codeofhonor.com/blog/whose-bug-is-this-anyway

See "Your computer is broken". They essentially inserted a stress test into the game that verified if the hardware was still doing calculations correctly, and if not, inform the user.

Re:ECC matters by Springcleaning on Sunday January 03, 2021 @02:43PM (#25624507)
Worse than a game crash is your data.

It is incomprehensible that there are still NAS devices being sold without ECC support.

Synology took a step in the right direction to offer prosumer devices with ECC but it is not really advertised as such. It is actually difficult to find which do have ECC and which ones don't.

Re:ECC matters by ksec on Sunday January 03, 2021 @04:23PM (#25625384)
>Synology took a step in the right direction to offer prosumer devices with ECC

I just look it up because if it was true it would have been news to me. Synology have been known to be stingy with Hardware Spec. But none of what I called Prosumer, the Plus Series have ECC memory by default. And there are "Value" and "J" Series below that.

Edit: Only two model from the new xx21 series using AMD Ryzen V has ECC memory by default.

Re:ECC matters by dannyw on Monday January 04, 2021 @03:14AM (#25628822)
Blame Intel for that. They dropped support for ECC RAM on the new Celerons and i3s.
Re:ECC matters by apankrat on Sunday January 03, 2021 @04:32PM (#25625454)
Aye. I have an assert in the code that fronts a very pedantic test of the context. In all cases when this assert was tripped (and reported) an overnight memtest86 test surfaced RAM issues.

- Edit -

Also, bit flips in the non-ECC memory are _the_ cause of the "bitrot" phenomenon. That is when you write out X to a storage device, but you get Y when you read it back. A common explanation is that the corruption happens _at rest_. However all drives from the last 30+ years have FEC support, so in reality the only way a bit rot can happen is if the data is damaged _in transit_, while in RAM, on the way to/from the storage media.

So, if you ever decide if to get an ECC RAM, get it. It's very much worth it.

Re:ECC matters by masklinn on Monday January 04, 2021 @10:46AM (#25632794)
> Also, bit flips in the non-ECC memory are _the_ cause of the "bitrot" phenomenon. That is when you write out X to a storage device, but you get Y when you read it back. A common explanation is that the corruption happens _at rest_. However all drives from the last 30+ years have FEC support, so in reality the only way a bit rot can happen is if the data is damaged _in transit_, while in RAM, on the way to/from the storage media.

Problems can definitely happen in the IO controller, RAID controller, cable, and disk controller. AFAIK all of these were seen and motivations for the existence of ZFS. One of their biggest insistence was that drives are universally lying bastards and should not be trusted any further than they can be thrown.

Re:ECC matters by srtjstjsj on Monday January 04, 2021 @12:01AM (#25628068)
Bitrot in human memory is the same. Memories change during the process of recalling them, not while they are in "storage".
Re:ECC matters by joshspankit on Monday January 04, 2021 @08:40AM (#25631157)
Hard to say for sure since we can’t measure “bit“ flips in-brain
Re:ECC matters by BlueTemplar on Sunday January 03, 2021 @01:57PM (#25624040)
Yeah, here's one example along many more :

https://forums.factorio.com/viewtopic.php?p=405060#p405060

Re:ECC matters by pkaye on Sunday January 03, 2021 @01:19PM (#25623663)
I wonder how much of those crashes are due to gamers aggressively overclocking their systems?
Re:ECC matters by lighttower on Sunday January 03, 2021 @04:28PM (#25625415)
Someone reads those reports!?! Wow, how do I write them to ensure someone who reads them takes them seriously?
Re:ECC matters by xmodem on Monday January 04, 2021 @01:23AM (#25628369)
The best way is to submit the same crash report from thousands of different locations, repeatedly
Re:ECC matters by faitswulff on Sunday January 03, 2021 @11:57AM (#25622936)
Do the crash reports include whether the machine has ECC memory?
Re:ECC matters by jacquesm on Sunday January 03, 2021 @12:04PM (#25623016)
On intel consumer boxes it is pretty safe to assume that they don't, on AMD it might be the case but it usually isn't.
Re:ECC matters by jackric on Sunday January 03, 2021 @12:03PM (#25623005)
Do the crash reports include recent solar activity?
Re:ECC matters by cbanek on Sunday January 03, 2021 @12:34PM (#25623258)
Well, I've had to actually worry about radiation bitflips as well. It does happen. But usually not so much on Earth!
Re:ECC matters by jgalentine007 on Sunday January 03, 2021 @01:06PM (#25623551)
One of the tire pressure sensors in my car tires had a bit flip a couple years ago and I had to reprogram it's ID. Luckily it was a subaru, so only a light came on in the dash.

My old Honda crv however would turn traction control on if your pressure was low - which worked by applying brakes to wheels that were slipping. If you were going up a slippery hill you would soon have no power, sliding backwards nearly off the road in nowhere West Virginia on the way to a ski resort.

Re:ECC matters by jjeaff on Sunday January 03, 2021 @02:51PM (#25624584)
How in the world would you ever know that problem was caused by a bit flip and not just one of the countless other reasons that a sensor could fail?
Re:ECC matters by jgalentine007 on Sunday January 03, 2021 @03:09PM (#25624779)
I have a TPMS programming tool (ATEQ QuickSet) and reader (Autel TS401), because I like to swap my winter / summer tires on my own. The TPMS light came on one day and inflating tires didn't help - I used the reader and found that one sensor's ID had changed. When I compared the ID (it was in hex) to the last programming - it was a single bit off. I couldn't reprogram the sensor itself, but I was able to update the ECU with the changed ID using the ATEQ.

I live in Denver but spend a lot of time skiing around 11k feet, maybe the higher elevation means more radiation.

Re:ECC matters by dharmab on Sunday January 03, 2021 @04:48PM (#25625565)
Similar story, we saw that one particular IP address in a public cloud network had a 3% TLS handshake error rate. We diverted traffic and then analyzed with wireshark. We found one particular bit was being pulled low (i.e. 0 -> 0 and 1 -> 0). HTTP connections didn't notice but TLS checksum verifications would randomly fail. Had a hell of a time convincing the cloud provider they had a hardware fault- turned out to be a bug which disabled ECC on some of their hardware.

Aside: I'm surprised you got a TPMS programming tool instead of a set of steelies. Big wheels? Multiple winter vehicles?

Re:ECC matters by jgalentine007 on Sunday January 03, 2021 @05:48PM (#25625988)
I have 2 cars. I like the TPMS to work since I've had 3 nails in tires in 4 years (newer construction area). Also the TPMS light in my impreza is almost as bright as the sun.
Re:ECC matters by cmehdy on Sunday January 03, 2021 @08:43PM (#25627123)
IIRC higher altitudes definitely register more radiation, the latitude effect being relatively minor in comparison.

Interestingly enough, average background radiation is actually the highest in the US exactly where you are[0] and I've seen comments (on a few websites talking about it) ranging from "the Rockies are loaded in Uranium" to "what about the Rocky Flats nuclear weapons site". Perhaps you're exactly in the right place to be an advocate for ECC :)

[0] http://www.radiationnetwork.com/

Re:ECC matters by amluto on Monday January 04, 2021 @12:54AM (#25628270)
Hah. My oven’s bottom unit stopped working. I strongly suspect a bit flip.
Re:ECC matters by dharmab on Sunday January 03, 2021 @12:39PM (#25623308)
I once got to tell a CTO the reason our shiny new point to point connection was suddenly trash was due to solar flares.
Re:ECC matters by jacquesm on Sunday January 03, 2021 @12:04PM (#25623017)
Timestamp + location should be enough to figure that out.
Re:ECC matters by ant6n on Sunday January 03, 2021 @12:27PM (#25623199)
It would be interesting to see whether there is a correlation between solar activity and game crashes — which in turn may provide an indication whether crashes are due to bugs or bit flips.
Re:ECC matters by joshspankit on Monday January 04, 2021 @08:43AM (#25631188)
As long as someone is tracking crash times/locations, the data is definitely there. Working tech support for major ISPs, there were some “surprising” large-queue days that correlated with solar flares.
Re:ECC matters by Triv888 on Sunday January 03, 2021 @12:04PM (#25623015)
most gaming desktops don't use ECC RAM anyways (at least those from a few years ago)