This is a simple example setup in a 12-drive enclosure. 9 disks, a global hot-spare disk and two cache drives.
Previously, I would have used straight ZFS mirrors for performance and expandability (raidz VDEVs can't be expanded), but at this scale, the 9-disk 3x3 raidz1 setup consistently outperformed a 10-disk RAID mirror setup in random and sequential I/O.
Re:ZFS High-Availability NAS
by eatstoomuchjam on Sunday August 14, 2016 @10:27AM
(#12285689)
A set of 5 stripes of mirrors has the same problem that the comment brought up - losing the _wrong_ two disks loses all of your data.
In this case, going to raidz2 across all 12 disks would give an extra disk worth of capacity/performance with the ability to lose any 2 disks without data loss and going to raidz3 should give the same capacity/performance with the ability to lose any 3 disks without data loss.
Any further failure would result in a complete data loss. Also, if any of the RAIDZ stripes loses more than one vdev simultaneously, this will lead to a complete data loss as well.