News for hackers. Stuff that matters.
Re:The State of ZFS on Linux by ryao on Thursday September 11, 2014 @02:26PM (#8304217)
A btrfs versus ZFS comparison probably deserves a blog post of its own, but I will try to address your question. I wrote the following on this topic last year:

https://groups.google.com/d/msg/funtoo-dev/g9OY_vqVpCM/VTKF8...

However, significant time has passed and it requires some corrections to be current:

1. I have not heard of any recent data corruption issues in btrfs, although I have not looked into them lately.

2. btrfs now has experimental RAID 5/6 support, but is neither production ready nor as refined as ZFS' raidz.

3. I should have said "inline block-based data deduplication". You can (ab)use reflinks to achieve a file-level data deduplication in btrfs, but it is not quite the same. btrfs now has a bedup tool that makes using reflinks somewhat easier now:

https://btrfs.wiki.kernel.org/index.php/Deduplication

4. btrfs now has some kind of incremental send/recv operation. However, it is not clear to me how it handles consistency issues from having "write-able snapshots":

https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

5. Illumos' ZFS implementation is now able to store small files in the dnode, which is improves its efficiency when storing small files in a manner similar to btrfs' block suballocation. This feature will likely be in ZoL 0.6.4.

Aside from those corrections, what I wrote in that mailing list email should still be relevant today. However, there are a few advantages that ZFS has over btrfs that I recall offhand that I do not see here there or in nisa's reply:

0. ZFS uses 256-bit checksums with algorithms that are still considered to be good today. btrfs uses checksum algorithms that are known to be weak. In specific, btrfs uses CRC32 on 32-bit processors and CRC64 on 64-bit processors. CRC32 is the same algorithm used by TCP/IP. Its deficiencies are well documented:

http://noahdavids.org/self_published/CRC_and_checksum.html

I have not examined CRC64, but I am not particularly confident in it. btrfs should have room in its on-disk data structures that would allow it to implement 256-bit checksums in a future disk format extension, but until then, its checksum implementations are vastly inferior.

1. The ztest utility that I described in the blog allows ZFS developers to catch issues that would have otherwise gone into production and debug them from userland. No other filesystem has something like quite like it.

2. ZFSOnLinux is the only kernel filesystem driver that is kernel version-independent, so if you are unable to upgrade your kernel, you can still get fixes. The inability of people to always update their kernels is an issue Linus mentioned at LinuxCon North America 2014.

3. The CDDL gives the ZFSOnLinux a patent grant for the ZFS patent portfolio. This is something that btrfs does not have and will likely never have unless Oracle decides to provide one. Consequently, Oracle is the only company in the world that I know is able to ship products incorporating the btrfs source code without being at risk should btrfs infringe on one of the dozens if not hundreds of patents in the ZFS patent portfolio. A small subset of them can be accessed from the Jeff Bonwick Wikipedia page:

https://en.wikipedia.org/wiki/Jeff_Bonwick

Re:The State of ZFS on Linux by cmurf on Friday September 12, 2014 @01:43AM (#8306728)
Metadata/data checksums are CRC-32C on all platforms and is per 4KB fs block. And ext4's (optional) checksumming also uses it. TCP is 16-bit and not CRC, although ethernet makes use of CRC-32. While SHA-2 is a cryptographic function and CRC-32C is not, and therefore inferior, as a checksum in the context of mostly but not entirely trusted hardware is adequate. It's also fast to the degree on modern hardware there's no point using nodatasum to disable it. Even 32-bit hardware handles it.

Btrfs send/receive snapshots are read-only. So you first take a read-only snapshot, send it and upon receive it's read-only. To make it rw, snapshot it (without -r), and optionally delete the ro snapshot.

I agree the kernel currency issue to get more urgent fixes is a problem for making filesystem usage friendly, in particular while development is heavy and backporting is non-trivial.

Re:The State of ZFS on Linux by ryao on Friday September 12, 2014 @09:35AM (#8307915)
You are correct. The ethernet frames use CRC32 while the IP frames use 16-bit CRC. While the checksums need not be of cryptographic quality (and not all checksum algorithms in ZFS are), there are multiple trivial ways in which the wrong data can match a CRC32 checksum and that is inadequate for the purposes of data integrity. It does help, but it does not go far enough.

As for my assertion that btrfs uses 64-bit CRC on 64-bit processors, this was the result of a misunderstanding between myself and the btrfs developers in #btrfs on freenode last year. I double checked with them and it turns out that you are correct that btrfs uses 32-bit CRC even on 64-bit processors. That makes btrfs much worse in this area than I previously thought.

Thanks for clearing some of the confusion about btrfs send/receive on its snapshots. To be clear, does this mean that btrfs will refuse to send a "writeable snapshot"?

Re:The State of ZFS on Linux by cmurf on Friday September 12, 2014 @02:54PM (#8309711)
I think the triviality and inadequacy are overstatements for the following two reasons: it's rare to have undetected errors in normally operating hardware that simultaneously go undetected by Btrfs. If this doesn't go far enough, then I'm challenged how to characterize the vast majority of the world's data on NTFS, HFS+, ext234, and XFS as being anything but fragile and tenuous, which would be an overstatement. What is going far enough? It's a use case question rather than something absolutely determinable. For a general purpose filesystem, I think CRC-32C is adequate, although it will be nice when there's choice for another algorithm to meet other use case requirements.

Yes, it won't send writable snapshots. e.g. ERROR: /mnt/root is not read-only.

Re:The State of ZFS on Linux by colin_mccabe on Friday September 12, 2014 @03:27AM (#8306921)
As others have commented, CRC32 or CRC64 being "weak" in a cryptographic sense doesn't mean it's not suitable for detecting disk errors. x86 has a crc32 instruction, making it far faster than any other option. By the way, TCP doesn't use CRC32... it uses a simple additive checksum.

Chris Mason wrote most of btrfs while he was at Oracle. Hence, the GPLv2 grants an implicit patent license to whatever patents Oracle might hold that bear on btrfs.

Re:The State of ZFS on Linux by ryao on Friday September 12, 2014 @09:32AM (#8307906)
Saying that a checksum is weak in a cryptographic sense would mean that it is possible for a malicious intelligence to generate collisions. This is not what I meant when I said that CRC was weak. Instead, I meant that it is trivial for ordinary glitches to corrupt data in ways that the checksums fail to detect. Things byte swaps and non-adjacent double bit flips are all that is necessary. The link I provided elaborates on this:

http://noahdavids.org/self_published/CRC_and_checksum.html

As for patent grants, you are thinking of the GPLv3. The GPLv2 does not provide any patent grant. That is one of the reasons why the GPLv3 was made. If you place code under the GPLv2, you do not necessarily provide a grant to any software patents to which you have rights that cover it. People who wish to use code encumbered by such patents must license it or be at risk a lawsuit.

Re:The State of ZFS on Linux by colin_mccabe on Thursday September 25, 2014 @11:29PM (#8371185)
As for patent grants, you are thinking of the GPLv3. The GPLv2 does not provide any patent grant.

This is incorrect. See http://en.swpat.org/wiki/GPLv2_and_patents

Re:The State of ZFS on Linux by ansible on Friday September 12, 2014 @11:58AM (#8308739)
This is not what I meant when I said that CRC was weak. Instead, I meant that it is trivial for ordinary glitches to corrupt data in ways that the checksums fail to detect. Things byte swaps and non-adjacent double bit flips are all that is necessary. The link I provided elaborates on this ...

Since we're talking about data corruption that has managed to not be detected by the hard drive's own firmware (a read error not detected as a read error), I now wonder what sorts of data errors might also slip by the 32-bit CRC used by btrfs.

It would be hilarious (in a bad way) if the same algorithm was used at both levels. So that the same byte swap (for example) would exactly bypass both the drive firmware and the check done by btrfs.

If the algorithms are different though, then I'd expect the chances are reduced that a particular error can slip by both.

Re:The State of ZFS on Linux by db48x on Monday September 15, 2014 @07:38PM (#8322219)
It's pretty scary; data on ordinary filesystems just evaporates over time. Undetected bit-flips every few terabytes from the drive itself, flaky cables and controllers, bit flips in your non-ecc ram, ghost writes, misaligned reads, firmware bugs, cheap port multipliers, etc, etc.

I've had a zfs filesystem for a few years now and twice it's detected and corrected an error that would have been silent data corruption in a lesser filesystem.

Re:The State of ZFS on Linux by colin_mccabe on Thursday September 25, 2014 @11:28PM (#8371184)
Serious servers use ECC RAM so bit-flips in memory are not an issue. CRC32 is also pretty robust against single-bit errors. If you think about it, you can't really make any guarantees for non-ECC systems. Flip the wrong bit, and the kernel will delete all your data. There is an if statement somewhere waiting to make this happen, and if the program code can randomly change, eventually it will.

Typical disk corruption patterns are a whole sector of zeroes being read, or reading data that belonged somewhere else. If you want to prove that ZFS's checksum is better, you need to prove that it's better against the patterns of corruption that actually occur.

Re:The State of ZFS on Linux by ScottBurson on Thursday September 11, 2014 @07:38PM (#8305850)
I got the impression somewhere that btrfs RAID 5/6 support allows, or will allow, new devices to be added to an existing RAID group.

That's an important feature for home and small business users. Having to replace every drive in a RAID group to grow it, as ZFS requires, is painful and expensive.

Fortunately, drives are cheap enough these days that you can just way oversize your pool to begin with. But anyone switching to ZFS should be aware of the need to do this.

Re:The State of ZFS on Linux by cmurf on Friday September 12, 2014 @01:51AM (#8306746)
You can add/delete devices from raid5/6 volumes now. The raid5/6 code is still experimental, in particular while detected problems are fixed on-the-fly to userspace, the fixes aren't written back to drives. That limitation applies to normal usage and scrubbing. A balance detects and fixes these.

Also the determination of a drive being "faulty" (in the md/mdadm sense) and how this gets communicated to userspace isn't in place. If it's in place for ZoL (?) that'd be a considerable difference, a bigger one than checksum algorithms in my opinion.

Re:The State of ZFS on Linux by voltagex_ on Thursday September 11, 2014 @11:52PM (#8306524)
Can you link to anything about oversizing the pool? I think I vaguely remember a trick with creating a sparse file and adding that to the pool until you can replace that with a drive.