While switching from Hardware Raid1 to ZFS recently, I had 1) sata cables failing on me and 2) a 8088-8087 PCI-Bracket corrupting half of the SATA connections through it. I would have never detected this without ZFS (scrub). All was fine for weeks. I did `fio` speedtests, `smart --long`, `dmesg` monitoring. Nothing. Until I ran the first scrub and 22 CRC read errors appeared in the 6x8TB RaidZ2. Much worse after switching drives and cables around. I spend like $400 for different cable-replacements, but finally identified the failing ones and all is good since then.

Now, I am more observant. Scrub once a month. Monitoring of SSD TBW, UDMA-CRC, Temperature, Watt, Fan-Speed in InfluxDB 2.0 with alerts set. Besides, ZFS mails all the important status info.

ZFS is really good for peace of mind, resilience and predictability, not necessarily for speed (but not exclusively, either), or your wallet.

Can you share what you use to capture the data (configs) and how you've configured your alerting?

I have written somee of it down here: https://du.nkel.dev/blog/2021-05-05_proxmox_influxdb/

Currently away, will update later.

It’s pretty remarkable actually just how bad that failure mode was. How did mdadm manage to cause such havoc after the broken device was reattached?

MDADM RAID1 does not seem to have the ability to recover from silent data corruption. In the standard two-disk RAID1, it's actually impossible to recover, since it doesn't know what copy is the uncorrupted one. With three disk RAID1 it would theoretically be possible to use a majority vote, but I don't believe MDADM has such code.

Btrfs raid1, on the other hand, uses the checksums of the data stored in its metadata trees to validate which copy of the data is correct and repair the corrupted copy. 3-disk makes this even more robust, as you get an exponential growth of data-metadata pairs to pull from. If any one pair matches, that data can be assumed to be correct.

Not sure how well this works out in practice, but if one drive returns a read error for a sector and the other returns the corresponding sector without an error, it should be a simple matter to trust the latter. Of course that assumes that storage corruption occurred after uncorrupted data was written to disk.

Modern drives have rather large per physical sector checksums to detect the occurrence of these errors and presumably retry a couple times before returning an I/O error if configured correctly.

In addition a good I/O bus should have error detection checksums for data in transit, and good hardware should have ECC RAM as well.

Indeed. That's why I specified /silent/ corruption. If the stack returns an explicit error, that is correctable (assuming the other disks don't error and return the correct data).

Sadly, a large number of SSDs (and nearly all cheap USB thumb drives / SD cards / eMMC devices) will happily return silently corrupted data.

The command told it to reattach the drive and they expected it to rebuild but instead md saw part of the mirror and simply added the drive as a good drive.

And the drive would return bad data instead of just erroring out - this is sometimes a difference between “home use” drives vs enterprise - enterprise assume you’re in a multi redundant raid setup and instead of retrying will just fail out fast and let the raid take care of it. Home drives will continue retrying and perhaps even eventually give you what they got - the idea being you’d rather have one byte bad of a word doc than lose a 4K chunk of it.

ZFS catches these tricks because the checksums will fail. Multi-redundancy RAID can do it - but since most are enterprise they are built with the assumption the drive will return good data - or none.

This can be the worst scenario as corruption can be silent and infect all backups after a time.

> The command told it to reattach the drive and they expected it to rebuild but instead md saw part of the mirror and simply added the drive as a good drive.

> ZFS catches these tricks because the checksums will fail.

Yea, I think this is the right explanation. I guess it simply doesn’t keep any consistency information that it can check, whereas ZFS keeps a checksum of every block. Having used ZFS, it is difficult for me to understand how anyone could trust anything else. I don’t think that there is anything a ZFS user could do to accidentally destroy their data (aside from obvious things such as deleting the wrong filesystem or pool, which could happen in any case).