A lot of what makes ZFS good is it's CoW implementation, that implementation is simplified by the concept that a block will never change. However the main underlying feature that would allow defragmentation is referred to as "block pointer rewrite. Effectively it would allow you to copy a block and apply any other transformation to it and then transactionally update all pointers to that block. This is very hard when you factor in all the things that could possibly point to a block including many snapshots/clones etc.
So the long and the short of it is the situation isn't great. Will we ever see BPR? Maybe. Is it still a really good filesystem even with this limitation? Definitely.
I've seen pools that have climbed above 90%, and maybe they've suffered permanent speed degradation, but they still run fast enough to mostly saturate a 10GbE connection, so... not a problem for me?
However, the effects scared us enough such that we will never let it happen again ... and that is a fairly severe economic and provisioning penalty (ie., chop off 15% of your inventory on top of the three drives per vdev you already lost for parity).
I searched that page (and this HN discussion) for the word "defrag" and got nothing. That's a problem.
Does the Oracle version of ZFS have defrag, or have it in the pipeline ?
It is not reasonable to expect folks to just never, ever, exceed 85% pool utilization. Further, most of those folks don't realize that there's no coming back from it - exceed 85% even for a day and you have a performance penalty forever.
"Oh it's no problem, just export the pool and recreate it"
Sure ... let's buy another $20k of equipment and duplicate a 250 TB pool just because we had a usage overrun (which wasn't really a usage overrun at all) one weekend.
Finally: think of the economics of this unofficial limit on your pools ... you probably aren't running a pure stripe, right ? Maybe you're running raidz3 ? So you already gave up three drives for data protection ... did your cost accounting also subtract another 15-20% from available storage space ?
ZFS needs defrag.