Thank you for the clarification - I had heard from a few sources that the block allocator algorithm actually changes at higher utilization, but was previously unable to find anything concrete in the documentation. This helped clear up a longstanding curiosity for me.

Does the problem go away again if you go back to below 80%?

After a bit of digging, yes but no.

So turns out it's a bit more involved than what's been commonly told as a straight up 80% == bad scenario. ZFS by default divides[1] each vdev (RAIDZ or mirror set) into ~200 allocation regions called metaslabs[2].

When allocating from a metaslab[3] it will check if the free space in that metaslab is below the threshold defined by metaslab_df_free_pct. It seems the threshold was changed to 4% free space at some point[4].

If the free space is above the limit it will use the fast first-fit search, if not it will use the expensive best-fit search.

However, as noted that threshold is per metaslab. So if the pool is fragmented, even though the overall free space in the pool is above the 4% threshold, there might be metaslabs with less than that free, which will lead to the expensive best-fit search.

So it's not a hard limit, but it should start to be noticeable above 80%.

[1]: https://www.delphix.com/blog/delphix-engineering/openzfs-cod...

[2]: http://dtrace.org/blogs/ahl/2012/11/08/zfs-trivia-metaslabs/

[3]: https://github.com/openzfs/zfs/blob/master/module/zfs/metasl... (note metaslab_df_free_pct)

[4]: https://www.truenas.com/community/threads/zfs-tweak-for-firs...

As you mention, it differs from system to system.

When it hits that threshold it becomes immediately noticeable - I/O performance will basically fall right down. At my old job we noticed it around 93% on Solaris 11.x, and maybe 85% on Solaris 10 off the top of my head, on two different production systems. Best practice was considered to keep pools on any ZFS system (OpenZFS or Solaris) below 80%.

Right, which is something I forgot to highlight.

If you have say a couple of 4TB disks in a mirror vdev, get it say 98% full, then expand adding new vdev with two 8TB disks mirrored, then you still have to pay the price whenever ZFS allocates from the first vdev.

ZFS tries to spread the load between vdevs in a weighted manner, which means most of the writes would go to the new vdev. But if it allocates from the first vdev you'll pay a latency price.

This means the latency in this case can be quite unpredictable, and not something that'll go away until enough blocks are freed from the first vdev.

Short answer, yes

I’ve read that any TRIM supported SSD prevents this and all SSDs have extra blocks (some more than others) that aren’t being utilized by default and are designed to replace any bad blocks. https://www.truenas.com/community/resources/some-differences... seems like ZFS might be special and if you have larger SSDs they will have different cell blocks and be faster because it’s 2x256 boards that can be used concurrently versus 1x256 which will have half the write speed. SSDs also complicate it further with RAM disks, mixed storage (like SLC+TLC/QLC) so the SSDs will be affected more if it’s a cheaper drive with no ram or SLC cache, and smaller sizes with less memory chips. I remember getting the Evo 850 because it had great firmware with ram and SLC cache, it was 3D TLC but it’s speed was still excellent.