Great reply :)
Your last two points are a little weak. Ultimately ZFS does more stuff, so on an SSD its going to be slower than a filesystem which doesn't have all these extra features if you aren't using them. I think there's a trade-off to be had.
All software has bugs, especially ZFS. You could argue it is easier to developer, test, and maintain individual device-mapper building blocks.
* There are issues when upgrading because the initramfs can store an old copy of the kernel module and the /dev/zfs interface is not stabilized. This will be addressed in the next 6 months by a combination of two things. The first is /dev/zfs stabilization. The second is bootloader support for dynamic generation of initramfs archives. syslinux does this, but it does not at this time support ZFS. I will be sending Peter Alvin patches to add ZFS support to syslinux later this year. Systems using the patched syslinux will be immune to this problem while systems using GRUB2 will likely need to rely on the /dev/zfs stabilization.
* There are many people who do not have problems, but this is certainly possible. Much of the weirdness should be fixed in 0.6.4. In particular, I seem to have fixed a major cause of rare weirdness in the following pull requests, which had the side benefit of dramatically increasing performance in certain workloads:
https://github.com/zfsonlinux/spl/pull/369 https://github.com/zfsonlinux/zfs/pull/2411
* The above pull requests have a fairly dramatic impact on NFS performance. Benchmarks shown to me by SoftNAS indicate that all performance metrics have increased anywhere from 1.5 to 3 times. Those patches have not yet been merged as I need to address a few minor concerns from the project lead, but those will be rectified in time for 0.6.4. Additional benchmarks by SoftNAS have shown that the following patch that was recently merged increases performance another 5% to 10% and has a fairly dramatic effect on CPU utilization:
https://github.com/zfsonlinux/zfs/commit/cd3939c5f06945a3883...
* There is opportunity for improvement in this area, but it is hard for me to tell what you mean. In particular, I am not certain if you mean minimum latency, maximum latency, average latency or the distribution of latency. In the latter case, the following might be relevant:
https://twitter.com/lmarsden/status/383938538104184832/photo...
That said, I believe that the kmem patches that I linked above will also have a positive impact on SSDs. They reduce contention in critical code paths that affect low latency devices.
Additionally, there is at least one opportunity to improve our latencies. In particular, ZIL could be modified to use Force Unit Access instead of flushes. The problem with this is that not all devices honor Force Unit Access, so making this change could result in data loss. It might be possible to safely make it on SLOG devices as I am not aware of any flash devices that disobey Force Unit Access. However, data integrity takes priority. You can test whether a SLOG device would make a difference in latencies by setting sync=disabled temporarily for the duration of your test. All improvements in the area of SLOG devices will converge toward the performance of sync=disabled. If sync=disabled does not improve things, the bottleneck is somewhere else.
* These alternatives operate on the block device level and add opportunity for bugs to cause cache coherence problems that are damaging to a filesystem on top. They are also unaware of what is being stored, so they cannot attain the same level of performance as a solution that operates on internal objects.