One of the biggest problems here is that spindles usually don't handle oversubscription well. They service requests quickly up to a certain queue depth, and after that I/O times become significantly longer and more erratic. So if you've got a RAID5 system swamped by a large streaming write workload, an fsync that triggers even a single write may take hundreds to thousands of milliseconds to complete.

ZFS does a few things here:

- On fsync(), the filesystem only records intent log records to disk. It doesn't record all the filesystem updates needed to implement the change. This cuts back on I/O. - ZFS supports using a separate physical device only for the intent log (called a "slog"). Even if a huge (non-synchronous) workload is slamming most of the spindles, writes to the slog can still complete quickly. This helps even if the slog is another spindle, but it's a huge win if it's an SSD. - ZFS throttles writes to avoid issuing so much to disks concurrently that they start behaving poorly. For more on this (and recent work to improve it), see http://dtrace.org/blogs/ahl/2013/12/27/zfs-fundamentals-the-....

This may all apply to ZFS on Linux as well.