Moving storage pooling functionality to the filesystem is the right call for anything with server workloads, ZFS's self-healing functionality wouldn't work without it (and this is precisely why I use FreeNAS instead Unraid or other alternatives, I care about the long-term integrity of my data, especially my VMWare images and photo collection).
#1: Do checksumming at the block level. Within each block, the RAID driver reserves a few bytes at the end for a checksum, and verifies them before returning data to the FS. (This would only work if the filesystem supports non-power-of-two block sizes.)
#2: Similar to #1, but pack the checksums for multiple data blocks into a separate dedicated checksum block. (This adds some extra read latency in the worst case, but caching could mitigate it.)
#3: Let the filesystem handle checksumming, but extend the block device API to provide feedback to the kernel if the checksum is invalid. I think you only really need two API calls -- one which says "read this block as fast as possible and I'll verify it", and one which says "the block you returned looks bad, try to reconstruct it and give me all available candidates".
This doesn't handle phantom or misdirected writes. You can't really do that unless you keep the checksums elsewhere, and specifically in the place where you intend to reference the data.
> Let the filesystem handle checksumming, but extend the block device API to provide feedback to the kernel if the checksum is invalid.
I think by the time you get this API right, it's going to seem like just as bad a layering violation.
ZFS has a coherent set of layers built into it. They're just not the ones that existed before it. But I don't think those previous layers were designed with many of the important failure modes of real hardware devices in mind. That's understandable given when they were designed, but we've come a long way since then in the understanding of those failure modes and our expectations of system integrity.
At the filesystem level, you can specify different redundancy policies for different types of data. You can mirror metadata but not file contents, for example, which is a pretty good policy available in BTRFS that you can't achieve at the block level. You can also do sensible things like mirror across three drives by putting each file on two of the three drives. But there's a better solution.
The better solution is to handle this at a higher level. It's common to want data replicated across machines, at which point block-level redundancy really only sucks up disk space for no benefit. This is how cloud storage works, and it's how you set up an Exchange server these days, et cetera.
RAID and block-level redundancy will still be around (somebody will always need it), but it's a dying technology, at least in the sense that new deployments are using it less and less frequently.
But these layers also impede a number of important things needed for data integrity checking and efficient rebuilding on failure, as well as complicating administration when you need to rebuild bits of an array after failure or fiddle with logical volumes. From the point of view of a sysadmin, ZFS is a revelation. Datasets are simple; creating, resizing (quota), deleting are trivial and safe. Operations on the pool are logical and (relatively) safer than with mdraid or hardware RAID. Snapshots are safer--you are doing it at the filesystem level rather than the block device level, so it can never be inconsistent or get corrupted when your snapshot device is invalidated at some arbitrary future point when the block delta exceeds the device size.
The layering is still there. It's just subtly different. If you're used to LVM, then a "pool" is basically a volume group. The "vdev"s making up the pool are the physical volumes, which might be RAID sets. The "datasets" or "zvols" are logical volumes. It's a reinvention of what we already had, but it's more powerful, more flexible, and vastly easier to administer.
I was sceptical, but after learning about it (and there is a learning curve), doing some simple test installs (often repeated as I realised I'd not set up the pool with the right blocksize or partition alignment, or the optimal dataset structure) and then some actual deployments, I'm a convert. What it offers is currently unmatched.
The btrfs FAQ [1] says that "unlike MD-RAID, btrfs knows what blocks are actually used by data/metadata, and can use that information in a rebuild/recovery situation", but is that really a good enough reason to reimplement the entire RAID subsystem from scratch? And couldn't TRIM/discard provide the same benefits?
For me, the clincher is that mdadm has a very stable, well-defined on-disk format, which is a huge bonus if you hit a bug or make a mistake. I once almost lost a personal RAID10 array by recklessly trying to add an extra drive with no backup, and without fully understanding what I was doing. I was able to recover all of my data by hacking together a Python script to reassemble all of the blocks in the correct order. I can't imagine how much effort that would have taken if I had to build something that understood the full details of btrfs's data structures.
[1]: https://btrfs.wiki.kernel.org/index.php/FAQ#Case_study:_btrf...