News for hackers. Stuff that matters.
Re:File Systems, Data Loss and ZFS by Someone on Friday September 19, 2014 @02:24PM (#8341749)
"In the case that we have two mirrored disks and accept the performance penalty of the controller reading both, the controller will be able to detect differences, but has no way to determine which copy is the correct copy."

If you 'seed' the checksum algorithm for a block with the block number being written, a subsequent read of a different block that produces the same data will have a checksum failure. That would make it possible to choose which block has the right data.

So, if you are willing to eat the performance, you can detect single misdirected writes.

Re:File Systems, Data Loss and ZFS by ryao on Friday September 19, 2014 @02:38PM (#8341832)
When I wrote that, I was talking about hardware RAID 1, which has no checksums.
Re:File Systems, Data Loss and ZFS by Someone on Friday September 19, 2014 @04:48PM (#8342409)
But the disks have their own checksums, haven't they?
Re:File Systems, Data Loss and ZFS by ryao on Friday September 19, 2014 @10:42PM (#8343512)
The low level formatting has ECC, which never leaves the drive. That said, there are two cases to consider for misdirected writes. One is that the write clobbers multiple sectors in which case you would get uncorrectable sectors. The second is that it perfectly replaces another sector. In that case, the ECC is a perfect match as the ECC is stored with the sector. Neither drive would report a problem, but the data would not match. This is what I described as being a problem and traditional RAID is incapable of dealing with it.
Re:File Systems, Data Loss and ZFS by Someone on Saturday September 20, 2014 @04:03AM (#8344008)
And that is where I stated that drives can report a problem. If they 'seed' their ECC algorithm with the sector number (XOR-ing the result with it would be sufficient), they can (statistically) detect that, when they read sector #X, what they got wasn't what they ever wrote as sector #X.

In fact, I guess they already do. If they didn't, there would be misdirected reads, too.

Re:File Systems, Data Loss and ZFS by ryao on Saturday September 20, 2014 @08:59AM (#8344393)
The low level formatting does include a sector number, but it is not part of ECC. I am not sure what your point is. Your theoretical description of how hard drives could work does not reflect reality. Research by CERN and others has confirmed the existence of misdirected writes. Deployed ZFS installations are detecting corruption in situations where the drives report everything is fine. Even if the storage hardware improves, having end to end checksums in the filesystem will continue to make sense.

That said, I think you are fixating on one way that things can go wrong. Another way that misdirected writes can occur is a bit-flip in the micro-controller's memory. This also allows for misdirected reads as well as reading/writing data that has a single bit flipped. These devices micro-controllers do not have ECC memory. Even if it were added, you still need to prove that there are no programming bugs via formal verification, but given that these devices are black boxes that cannot be inspected, you cannot rely on the claim of a proof even if one is done and there would still be the possibility for errata in the micro-controller. It is far easier to just use end-to-end checksums in the filesystem. Even if you think the device is trustworthy, end-to-end checksums give you the ability to check that it is doing what it is supposed to do. You simply do not have that with traditional RAID.