Why is ZFS better (IMHO):
+ Tooling. The zpool and zfs commands are clear, easy to use and well documented. btrfs e.g. has no way to get a list of files with checksum problems. Also automounting and setting flexible mount points as well as the whole zfs get/set concept felt like a really concise way to configure all aspects easily.
+ Architecture. ZFS uses merkle hash trees with multiple copies spread over the disk (ditto blocks). This means that disk failures likely won't affect metadata and every possible disk problem is accounted for (except main memory). btrfs only uses crc32c and has 2 copies of the metadata tree on disk (but not spread over the disk afaik) or simple RAID1 in case of multiple disks. So in case of disk problems btrfs tends to corrupt faster. I've actually tested this and it's almost impossible to corrupt ZFS metadata even with hundreds of bad blocks. btrfs tends to switch faster to read-only mode with bad disks.
+ Compression: LZ4 is slightly better than LZO
+ L2ARC/ARC - easy to plug in a SSD cache. ARC is better than the pagecache for most workloads.
+ It's really fun to work with. A lot of good documentation. Dealing with btrfs felt very different...
+ RAIDZ(2-3) and copies=n for datasets. Both don't exist in btrfs (yet).
Reasons against ZoL:
- Memory. I've got out of memory errors because ARC memory is not freed fast enough. Also ZFS hogs a lot of memory. At least 750MB to 1GB kernel memory with several disks. btrfs needs far less. There is upcoming work to fix the ARC out of memory issues through.. it's on the roadmap from the developers.
- Stability. This also applies to btrfs but it's not as rock-solid as ext4. Especially with broken disks and usage of advanced features you'll run eventually into bugs. But the "normal" operations where 100% stable for me with ZoL.
- It's not native Linux. You have some Solaris Porting layer modules and integration into the Linux kernel is not as tight as you may like it to be. E.g. cgroups based io throttling was not possible, last time I looked. This may change in the future but it's work to integrate it and if e.g. the module does not not compile for a new kernel you are screwed.
- btrfs will be supported and be a default choice in 1-2 years in likely all major Linux enterprise distributions. SLES already made it the default. For most workloads ZFS is likely not worth it. At least it's tough to sell against a default option.
- rootfs on a ZoL volume is quite a lot of work and difficult to get working. But this is largely a distribution problem.
To expand on your LZ4 comment, LZ4 is 3 times faster than LZO at decompression:
https://code.google.com/p/lz4/
However, LZ4's real innovation is on incompressible data. I do not have a reference at the moment, but there have been benchmarks on the userland lz4 tool on incompressible data that show it processing 10GB/sec. This is because LZ4 uses a hash table that enables it to give up when trying to compress data far sooner than other lempel-ziv implementations do.
As for stability, it is hard to for me to tell what you mean by that, but I just wrote a series of blog posts on the topic. If there are any outstanding issues not listed there, please file an issue on github:
https://github.com/zfsonlinux/zfs/issues/new
I also encourage you to file an issue requesting support for cgroup bsaed IO throttling. You are the first person to mention it.
As for not compiling with new kernels, the 0.6.3 release supports Linux 2.6.26 to Linux 3.16 while HEAD adds support for Linux 3.17 release candidates.
I briefly touched on the rootfs on ZFS issue in my blog post. I plan to add ZFS support to syslinux in the near future as per a discussion that I had with Peter Alvin at LinuxCon North America 2014. I expect this to resolve the issue for those willing to use syslinux as their bootloader.
As for cgroups blkio - there is already a bug report: https://github.com/zfsonlinux/zfs/issues/1952
Sorry. It was not my intention to suggest that ZFS does not compile on new kernel versions. I've never had problems with that. I just wanted to point out that it's not in the mainline kernel and for some people that might be important.
Thanks for your great work on ZoL! It made my life a lot easier :)
As for the issues you linked, the project will likely not be able to tackle them until next year, but being in the tracker means that they will receive attention.
WRT to Gentoo, Gentoo does have a tool called genkernel that I consider to make such upgrades fairly easy. It can handle any of initramfs generation, kernel compilation, kernel configuration and bootloader configuration (GRUB2). However, it does have a moderate learning curve. I am also one of the genkernel developers, so I likely have an implicit advantage over users in understanding genkernel. In particular, I find some users have difficulty discovering its functionality, despite it having things fairly well documented in the man page. There is likely more that needs to be done in this area.
https://groups.google.com/d/msg/funtoo-dev/g9OY_vqVpCM/VTKF8...
However, significant time has passed and it requires some corrections to be current:
1. I have not heard of any recent data corruption issues in btrfs, although I have not looked into them lately.
2. btrfs now has experimental RAID 5/6 support, but is neither production ready nor as refined as ZFS' raidz.
3. I should have said "inline block-based data deduplication". You can (ab)use reflinks to achieve a file-level data deduplication in btrfs, but it is not quite the same. btrfs now has a bedup tool that makes using reflinks somewhat easier now:
https://btrfs.wiki.kernel.org/index.php/Deduplication
4. btrfs now has some kind of incremental send/recv operation. However, it is not clear to me how it handles consistency issues from having "write-able snapshots":
https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
5. Illumos' ZFS implementation is now able to store small files in the dnode, which is improves its efficiency when storing small files in a manner similar to btrfs' block suballocation. This feature will likely be in ZoL 0.6.4.
Aside from those corrections, what I wrote in that mailing list email should still be relevant today. However, there are a few advantages that ZFS has over btrfs that I recall offhand that I do not see here there or in nisa's reply:
0. ZFS uses 256-bit checksums with algorithms that are still considered to be good today. btrfs uses checksum algorithms that are known to be weak. In specific, btrfs uses CRC32 on 32-bit processors and CRC64 on 64-bit processors. CRC32 is the same algorithm used by TCP/IP. Its deficiencies are well documented:
http://noahdavids.org/self_published/CRC_and_checksum.html
I have not examined CRC64, but I am not particularly confident in it. btrfs should have room in its on-disk data structures that would allow it to implement 256-bit checksums in a future disk format extension, but until then, its checksum implementations are vastly inferior.
1. The ztest utility that I described in the blog allows ZFS developers to catch issues that would have otherwise gone into production and debug them from userland. No other filesystem has something like quite like it.
2. ZFSOnLinux is the only kernel filesystem driver that is kernel version-independent, so if you are unable to upgrade your kernel, you can still get fixes. The inability of people to always update their kernels is an issue Linus mentioned at LinuxCon North America 2014.
3. The CDDL gives the ZFSOnLinux a patent grant for the ZFS patent portfolio. This is something that btrfs does not have and will likely never have unless Oracle decides to provide one. Consequently, Oracle is the only company in the world that I know is able to ship products incorporating the btrfs source code without being at risk should btrfs infringe on one of the dozens if not hundreds of patents in the ZFS patent portfolio. A small subset of them can be accessed from the Jeff Bonwick Wikipedia page:
Btrfs send/receive snapshots are read-only. So you first take a read-only snapshot, send it and upon receive it's read-only. To make it rw, snapshot it (without -r), and optionally delete the ro snapshot.
I agree the kernel currency issue to get more urgent fixes is a problem for making filesystem usage friendly, in particular while development is heavy and backporting is non-trivial.
As for my assertion that btrfs uses 64-bit CRC on 64-bit processors, this was the result of a misunderstanding between myself and the btrfs developers in #btrfs on freenode last year. I double checked with them and it turns out that you are correct that btrfs uses 32-bit CRC even on 64-bit processors. That makes btrfs much worse in this area than I previously thought.
Thanks for clearing some of the confusion about btrfs send/receive on its snapshots. To be clear, does this mean that btrfs will refuse to send a "writeable snapshot"?
Yes, it won't send writable snapshots. e.g. ERROR: /mnt/root is not read-only.
Chris Mason wrote most of btrfs while he was at Oracle. Hence, the GPLv2 grants an implicit patent license to whatever patents Oracle might hold that bear on btrfs.
http://noahdavids.org/self_published/CRC_and_checksum.html
As for patent grants, you are thinking of the GPLv3. The GPLv2 does not provide any patent grant. That is one of the reasons why the GPLv3 was made. If you place code under the GPLv2, you do not necessarily provide a grant to any software patents to which you have rights that cover it. People who wish to use code encumbered by such patents must license it or be at risk a lawsuit.
This is incorrect. See http://en.swpat.org/wiki/GPLv2_and_patents
Since we're talking about data corruption that has managed to not be detected by the hard drive's own firmware (a read error not detected as a read error), I now wonder what sorts of data errors might also slip by the 32-bit CRC used by btrfs.
It would be hilarious (in a bad way) if the same algorithm was used at both levels. So that the same byte swap (for example) would exactly bypass both the drive firmware and the check done by btrfs.
If the algorithms are different though, then I'd expect the chances are reduced that a particular error can slip by both.
I've had a zfs filesystem for a few years now and twice it's detected and corrected an error that would have been silent data corruption in a lesser filesystem.
Typical disk corruption patterns are a whole sector of zeroes being read, or reading data that belonged somewhere else. If you want to prove that ZFS's checksum is better, you need to prove that it's better against the patterns of corruption that actually occur.
That's an important feature for home and small business users. Having to replace every drive in a RAID group to grow it, as ZFS requires, is painful and expensive.
Fortunately, drives are cheap enough these days that you can just way oversize your pool to begin with. But anyone switching to ZFS should be aware of the need to do this.
Also the determination of a drive being "faulty" (in the md/mdadm sense) and how this gets communicated to userspace isn't in place. If it's in place for ZoL (?) that'd be a considerable difference, a bigger one than checksum algorithms in my opinion.
[1]: Seen that 90 drive Supermicro drive chassis[2]? Three 29-drive RAID-Z3s with 3 hot spares (which is shared across the three RAIDs) in a single storage pool (round robin-esqued), and you could plug that into a 2U with 16 small SSDs (like those Crucial M550 128GBs I use now) for the ZIL/L2ARC farming, and then have, well, near-infinite IO performance. Good luck trying to assemble that with BTRFS.
[2]: A wet dream for CEPH users, too.
There is a tradeoff of, for example, using 3 RAIDZs each with a third of the drives in your pool, or 1 RAIDZ3 using all the drives in your pool.
In the first case, a rebuild takes less time, but the chances of all data in that RAIDZ being lost is (I think) 4 times higher (but statistics is not my strong point; it may be higher).
In the second case, a rebuild takes longer, the chances of another drive failing before rebuild is complete is higher, however, you require three failures before you have a problem; the array is still fully protected from another drive failure after two drive failures.
The numbers in the example I gave, however, are not quite optimum for performance reasons (you'd be better off with 16 or 24), but most people don't like tons of hotspares (even though that case likely eats drives like old Sun thumpers would).
All of that said, it depends entirely on how much you want data reliability. And, also, it was just an example. Not necessarily a good one. I was just illustrating that ZFS can handle that many drives with ease, but BTRFS would not be a good fit.
[1]: There seems to be no difference between power of two and half way steps in real world performance when the number of drives are big enough, as in, 8, 12, and 16 will all perform similarly.
At work where we use ZFS extensively we more or less use # of vdevs * I/O performance of a single drive = total "worst-case" IOPS prior to caching. Caching of course is where ZFS will start to shine - it will win no performance crowns when put up against a "traditional" RAID6 array with the same number of spindles.
This is also why most production implementations of ZFS utilize many sets of mirrors. For example your 90 drive example would be 45 sets of mirrors in our environment, granting roughly 45*120 iops (if NL SAS) total peak capacity. We avoid RAIDZ largely for the reason it would drastically lower the total # of vdevs, and performance falls off a cliff in that situation.
ZFS (not on Linux) seems to handle up to around 300 drives fairly handily before you start to run into issues I've found. 280 is our current max cluster design.
However, in cases where you are using SSDs for ZIL/L2ARC, vdev spam isn't as big of a factor. In the example I gave I was illustrating the amount of usable storage you could get in the context of having it reliable, not comparing performance of RAID-Z vs non-ZFS RAID.