The truth is that everyone should want to use ECC wherever data integrity is a priority, regardless of whether or not they’re using ZFS. The conjecture about ZFS being uniquely vulnerable has been debunked over and over again. Maybe there was something different back in 2014 when this was written, but even if that was the case it’s not a good idea to use such old blog posts full of admitted conjecture to anchor your technology understandings.
The only things that really happened was that a) ZFS exposes how failure prone hardware is due to pervasive checksumming b) ZFS came from culture (and project goals) that prioritized data safety - as such, it was possibly first contact for many with strong recommendations about ECC memory
The amount of people rehashing this "you must use ECC with ZFS" was insane.
One guy seemed to have made it his life's mission to publish this same story over and over again.
I think the misunderstanding goes something like this. I'm pretty sure all three of the below statements are true:
• People who prioritize data integrity are more likely to be using ZFS.
• People who prioritize data integrity should use ECC memory.
• Ergo, if you are using ZFS, you should probably be using ECC memory.
However, it doesn't naturally follow that because you are using ZFS, you should be using ECC memory, nor that people without ECC memory should not use ZFS. This is unintuitive.
I use a usb-attached zfs raid-1 for my important storage. I have to run "zfs-fuse" before I mount it.
What I know about ecc comes from this:
https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...
I have never scrubbed my zfs vault, but I have scrubbed my btrfs drives.
I should get something with ecc. Wow, do I have a lot of stuff.
I have no idea why one would use a 2006 Google summer of code project instead of the official openzfs project which provides an actual kernel module that would be several times faster and likely safer.
The problem was the same even in 2014: ZFS lets you know how shitty your hardware is.
And, just like 2014, people would rather shoot the messenger because they don't have any power to force companies to make better hardware.
If you're Google (you're not Google) then the scale of errors becomes a choice of optimization.
The frequency of DRAM errors is proportional with the amount of memory installed in a computer. For smaller quantities of memory, e.g. 16 GB or 32 GB, the frequency might be of one error every few months. With ECC, the mean time between errors is likely to become longer than the lifetime of the computer.
Intel, who has initiated this absurdity of convincing the naive customers that it is OK to buy computers that may make mistakes from time to time, has succeeded to get away with it only due to the huge amount of software bugs, which have made the computer users believe that whenever a computer crashes, or some corrupted data is discovered, it is more likely that the cause has been a software bug and not a hardware defect.
The integrated circuits made with modern CMOS processes, with very small devices, have a non-negligible aging rate.
Because of that, memory modules that have been used for many years start at some point to have much more frequent errors than when new.
When you have ECC, such memory modules are immediately detected and they can be replaced before causing some irremediable data corruption. This helped me a lot a few times.
It's some sort of variation of the Toupee Fallacy, you aren't aware of the frequency of bit errors because you have absolutely no way of knowing when they happen, and what "random" errors were caused by them.
I don’t care about triggering an occasional bug or a bit flip in a video stream, these things are inconsequential. If a bit gets flipped in a photo it’ll be a very slightly different photo. A piece of text will be slightly different… if either had the luck to be in memory ready to write when hit, in dozens of gigabytes of memory.
The most likely consequence of a bit flip is nothing at all happening because it hit unallocated memory. The next most likely consequence is nothing as it hit a piece of program memory that has no effect on operation and gets overwritten in moments.
ECC zealots are really excited about preventing something that just doesn’t happen all that often: actual data corruption with permanent significant effects.
This is really weird. How come even routers have ECC memory while our actual computers don't?
Why can't all memory just be ECC memory? I don't even see the point of non-ECC memory. Everything has error correction built-in: hard drives, SSDs, even optical media. Yet RAM doesn't?
(The 1st generation of Pentium, the 60/66 MHz models, had been introduced in 1993, while the 2nd generation, the 90/100 MHz models, introduced in 1994, used a different socket and different motherboards.)
Unlike with the previous Intel CPUs, Intel had the idea to introduce a market segmentation between Pentium and Pentium Pro (for the successors of Pentium Pro, Intel created the Xeon brand, a couple of years later).
Therefore, Intel has retained the ability to detect and or correct memory errors only in the Pentium Pro chipsets (at that time the memory controllers were in the so called northbridge, a part of the chipset, and not in the CPU), while removing this feature from the Triton chipset intended for Pentium.
While Intel has initiated this market segmentation policy, between "amateurs" and "professionals" (unlike IBM, which since the first IBM PC had always included memory error detection via parity), both the memory module vendors and the makers of other CPUs have been also happy to follow this policy, because it allows them to extract much more money from the knowledgeable users than the cost of adding ECC, while also having larger profit margins when selling to naive users, because the elimination of ECC/parity has never caused any price reduction.
When this feature was removed, all the new motherboards and modules without parity/ECC had the same prices as the previous models with error detection (for many years, the memory errors were detected via parity, but then the memory controllers were improved, so that when using the same number of extra bits, i.e. the same memory modules and motherboards as for parity, single errors could be corrected, not only detected).
It's as suspect as I have nothing to substantiate it but my kid caught autism after that shot.
It's so badly reasoned it calls into question the authors judgment on any technical topic because it says his basic tools for reasoning are lacking.
99.9% of data loss occurs because of well understood but difficult to fully mitigate problems.
There is no reason to believe zfs is more susceptible to bad memory and it's not exactly new having begun development 21 years ago.
ECC is corruption of the parts in memory, like caches of data, data when it's in the process of being compressed or encrypted, some management data always cached in memory etc.
In the end not having ECC can affect your system in such abstruse ways that IMHO it probably doesn't make any relevant difference if you use zfs or e.g. ext4.
Tbh it's kind absurd that ECC is not the standard for any non-"cheap" PC/Laptop.
EDIT: Checksums can still potentially help in the unlikely case that the RAM error is a bitflip in some loaded offset/ptr referring "into" the HDD/SSD storage (e.g. position of some file on disk).
No, it won't.
https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y...
> Let’s assume that we have RAM that not only isn’t working 100% properly, but is actively goddamn evil and trying its naive but enthusiastic best to specifically kill your data during a scrub. First, you read a block. This block is good. It is perfectly good data written to a perfectly good disk with a perfectly matching checksum. But that block is read into evil RAM, and the evil RAM flips some bits. Perhaps those bits are in the data itself, or perhaps those bits are in the checksum. Either way, your perfectly good block now does not appear to match its checksum, and since we’re scrubbing, ZFS will attempt to actually repair the “bad” block on disk. Uh-oh! What now?
> Next, you read [a copy of the same block from another disk]. Now, if your evil RAM leaves this block alone, ZFS will see that the second copy matches its checksum, and so it will overwrite the first block with the same data it had originally – no data was lost here, just a few wasted disk cycles. OK. But what if your evil RAM flips a bit in the second copy? Since it doesn’t match the checksum either, ZFS doesn’t overwrite anything. It logs an unrecoverable data error for that block, and leaves both copies untouched on disk.
If there is a filesystem that is dumb enough to cause corruption during the checksumming process, please let me know which one, so I can be sure to never ever ever go anywhere near it. :)
Error checking will only ever help you, not hurt you. It doesn’t matter how bad you memory or disk or raid controller is. Error checking won't necessarily save you from those things, but it can in some cases, and it’ll never make things worse.
"There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS."
Wow. If you weren't talking about ZFS in a comment under an article concerning ZFS, an article concerning the same issue you are parroting, an issue that supposedly effects ZFS, what filesystem were you talking about?
Let me guess, a hypothetical filesystem.
Sure, whatever.
> bad memory corrupting data being actively changed remains a problem for any filesystem. If the filesystem is actively changing data it can not rely on anything it is writing to disk being correct if the in memory buffers and other data structures are themselves corrupted.
And, yet, that's not the argument you just made. This is what makes me thinks this is a bad faith pivot to something, anything respectable. It's not like I'm going to forget that you said right above (it's still there!) that a scrub could destroy all your data. This claim has been repeatedly debunked. You may have been taken in. Fine. That happens. Just, now that you know, don't keep spreading this FUD.
And zfs uses more and relies on it more than others, above that baseline.
> There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS.
And goes on to say
> I have nothing to substantiate this, but my thinking is that since ZFS is a way more advanced and complex file system, it may be more susceptible to the adverse effects of bad memory, compared to legacy file systems.