Re:We saved millions in SSD costs by upgrading our filesystem
by magicalhippo on Wednesday November 10, 2021 @08:41AM
(#29174502)
Right, which is something I forgot to highlight.
If you have say a couple of 4TB disks in a mirror vdev, get it say 98% full, then expand adding new vdev with two 8TB disks mirrored, then you still have to pay the price whenever ZFS allocates from the first vdev.
ZFS tries to spread the load between vdevs in a weighted manner, which means most of the writes would go to the new vdev. But if it allocates from the first vdev you'll pay a latency price.
This means the latency in this case can be quite unpredictable, and not something that'll go away until enough blocks are freed from the first vdev.
If you have say a couple of 4TB disks in a mirror vdev, get it say 98% full, then expand adding new vdev with two 8TB disks mirrored, then you still have to pay the price whenever ZFS allocates from the first vdev.
ZFS tries to spread the load between vdevs in a weighted manner, which means most of the writes would go to the new vdev. But if it allocates from the first vdev you'll pay a latency price.
This means the latency in this case can be quite unpredictable, and not something that'll go away until enough blocks are freed from the first vdev.