You can use different sized drives in ZFS, but you will be limited to the utilization of the lowest capacity drive on the larger drives in the pool. In practice this isn't much of a problem, because you can just replace the lowest capacity drives first to grow your pool size over time.
Small, but important, correction: the drive size limiting is per vdev. A pool consist of one or more vdevs.
So say you have a pool consisting of two mirrored vdevs, one with a pair of 3TB and one with a pair of 4TB drives, and one 3TB drive dies. If your local supplier don't have the same 3TB disk in stock, you can take a 4TB or similar and pair it with the old 3TB.
The pool will then still have a capacity of 3+4 = 7TB, but at least each mirrored pair will be intact.
That is true, however you can be somewhat more flexible if you go for plain mirroring. Then "only" pairs of disks need to be the same size.
Until not too long ago it wasn't advised to have wildly different sized drive-pairs in the same pool, but IIRC some work was done which alleviates this concern.
Mirroring also allows for more affordable upgrades in the sense that you "only" need to to replace a single pair with bigger drives if you want to increase capacity without adding more drives.
However watching his presentation about RAID-Z expansion I couldn't shake the feeling that that is a rather arbitrary decision. I mean, if you take his slide visualizing the reflow, it seems that as long as you can put the data + parity blocks on different devices, the size of each device in the vdev could be different.
Of course if you only write full stripes (ie one block for each disk) then you would still be limited by the smallest disk, however the beauty is that you can write shorter stripes. So couldn't ZFS just spread those blocks out?
Though it's late, so maybe I missed something obvious here :)
I have two such setups since 2016 or so, and they haven't given me any problems.
Do you need the performance or capacity? What are your needs? How many drives can your chassis hold. How much space do you need?
And so on. Maybe RAID10 is the best option for you but it's not a straight rule-of-thumb.
That is also true for RAID 5/6 (since all drives are stressed during a resilver of a RAID 5/6).
> RAID6 is safer: any two drives can fail. RAID 10 can sustain more drive failures, but can tolerate only 'certain' drives to fail
You should consider two-way mirrors as having the same redundancy as RAID5, but with faster resilver times (a byte-for-byte copy is faster than parity calculations -- and you should want your pool to be in a degraded state for as short a time as possible). You might survive more disk failures, but you shouldn't count on it.
If you want RAID6-like redundancy, use 3-way mirrors. But that's obviously more expensive than RAID6 (even though I would strongly argue that 3-way mirrors are far safer than RAID6).
For RAID10 to fail, two disks need to fail, and they need to be a mirrored pair, so it's a conditional probability. It's possible to lose up to n/2 disks and for the array to stay up.
For RAID6 to fail, three disks need to fail, but once three disks fail, that's it, you're out.
This all means that whether RAID6 is better than RAID10 is dependent on the number of disks and the actual failure rate. The more disks you have in your array, the more likely RAID10 is to be safer than RAID6.
RAID10 is much, much safer than RAID5. They are not similar in reliability.
Most of the time, RAID6 is safer than RAID10, but RAID10 gets you a lot better performance.
My understanding is that ZFS forces you to be a lot less flexible with which size drives you put into the array, which is what attracted me towards BTRFS in the first place. Aside from Raid 5/6 configurations, I've never had someone tell me to explicitly avoid BTRFS.
But whenever I see any articles about Raid, they're always using ZFS.
It's gotten noticeable enough to me that I'm left wondering if the general consensus is that everyone should just be using ZFS, even if that's something that I don't see spoken out loud very often.