Re:ZFS: Use mirror vdevs, not RAIDZ
by aidenn0 on Wednesday December 09, 2020 @12:00PM
(#25360754)
I was running 3x 12 drive vdevs in raidz2 and write performance was terrible. We moved some data to a different machine and rebuilt as 18 mirrors. This was a long time ago (as in running on Solaris long time ago), so maybe things are better now.
[ Parent
| on HN ]
Re:ZFS: Use mirror vdevs, not RAIDZ
by segfaultbuserr on Wednesday December 09, 2020 @04:47PM
(#25365677)
Slow performance is always my impression with RAIDZ. There's an obvious performance hit if you are coming from conventional RAID setups (like mdadm). I've even seen some unbelievable slow speed like 1 MiB/s in the middle of copying a large git repo (which have a lot of small files, repeated git pull also creates a lot of fragmentation). But my experience was based on early ZFSOnLinux, perhaps a real BSD will perform better. Anyway, I knew ZFS's first goal is data safety, not performance, and it's why I could tolerate the performance...
Each 12 drive array would have an interesting physical block layout.
What was your ASHIFT and what was your physical disk raw sector size because if you had a ASHIFT=9 you would have 5120k blocks, if it was a ASHIFT=12 you would have physical 40960 blocks. None of those line up well with zfs.
Additionally if this was a long time ago the pool would have defaulted to ASHIFT=9 and if you added any 4K drives you could have been having write amplification occurring.
Looks as if you had your drives doing a lot of unnecessary extra wasted work.
This is huge thing (called "Slop Space") that I noticed after I bought all ZFS NAS parts!
It wastes much capacity by default if drive count is not ideal. IIRC use larger recordsize (1MB or above) and enable compression almost solves issue. I use default 128K recordsize for small files zvol for performance, and use larger recordsize for the rest of zvols.
Ashift=9 native 512b sector drives