MDADM RAID1 does not seem to have the ability to recover from silent data corruption. In the standard two-disk RAID1, it's actually impossible to recover, since it doesn't know what copy is the uncorrupted one. With three disk RAID1 it would theoretically be possible to use a majority vote, but I don't believe MDADM has such code.

Btrfs raid1, on the other hand, uses the checksums of the data stored in its metadata trees to validate which copy of the data is correct and repair the corrupted copy. 3-disk makes this even more robust, as you get an exponential growth of data-metadata pairs to pull from. If any one pair matches, that data can be assumed to be correct.

Not sure how well this works out in practice, but if one drive returns a read error for a sector and the other returns the corresponding sector without an error, it should be a simple matter to trust the latter. Of course that assumes that storage corruption occurred after uncorrupted data was written to disk.

Modern drives have rather large per physical sector checksums to detect the occurrence of these errors and presumably retry a couple times before returning an I/O error if configured correctly.

In addition a good I/O bus should have error detection checksums for data in transit, and good hardware should have ECC RAM as well.

Indeed. That's why I specified /silent/ corruption. If the stack returns an explicit error, that is correctable (assuming the other disks don't error and return the correct data).

Sadly, a large number of SSDs (and nearly all cheap USB thumb drives / SD cards / eMMC devices) will happily return silently corrupted data.

The command told it to reattach the drive and they expected it to rebuild but instead md saw part of the mirror and simply added the drive as a good drive.

And the drive would return bad data instead of just erroring out - this is sometimes a difference between “home use” drives vs enterprise - enterprise assume you’re in a multi redundant raid setup and instead of retrying will just fail out fast and let the raid take care of it. Home drives will continue retrying and perhaps even eventually give you what they got - the idea being you’d rather have one byte bad of a word doc than lose a 4K chunk of it.

ZFS catches these tricks because the checksums will fail. Multi-redundancy RAID can do it - but since most are enterprise they are built with the assumption the drive will return good data - or none.

This can be the worst scenario as corruption can be silent and infect all backups after a time.

> The command told it to reattach the drive and they expected it to rebuild but instead md saw part of the mirror and simply added the drive as a good drive.

> ZFS catches these tricks because the checksums will fail.

Yea, I think this is the right explanation. I guess it simply doesn’t keep any consistency information that it can check, whereas ZFS keeps a checksum of every block. Having used ZFS, it is difficult for me to understand how anyone could trust anything else. I don’t think that there is anything a ZFS user could do to accidentally destroy their data (aside from obvious things such as deleting the wrong filesystem or pool, which could happen in any case).