do you have lots of snapshots? every snapshotting FS I've worked with has really slow deletes, especially when the volume is near capacity.

Snapshots are one thing ZFS is fast at. All the blocks for a given snapshot are placed on a "deadlist". Snapshot deletion is essentially just returning this list of blocks back to the free pool. A terabyte snapshot will take a short while (in the background) to recycle those blocks. But the deletion itself is near instantaneous.

I think you misunderstand: file deletions are what is slow (I don't use ZFS, my reference is WAFL, but my understanding is that all snapshotting file systems have this problem).

Even this should have minimal overhead. If the file is present in the snapshot, then it's simply moving the blocks over to the deadlist which is a very cheap operation. If it's not in the snapshot then the blocks will get recycled in the background. In both cases you should have the unlink complete almost immediately.

All of the snapshot functionality is based upon simple transaction number comparisons plus the deadlist of blocks owned by the snapshot. Only the recycling of blocks should have a bit of overhead, and that's done by a background worker--you see the free space increase for a few minutes after a gargantuan snapshot or dataset deletion, but the actual deletion completed immediately.

I've been promised many things by vendors and they always fall back to "hey! look! cool CS file system theory". I test my systems carefully and report the results back; they often don't agree.

I should point out again that I don't have enough direct experience with ZFS to say if this is the case, my experience was with an enterprise NetApp server at a large company that was filling the disk up (>95%) in addition to doing hourly snapshots.

I have 400 in total, though none on the slow volume :/ That shouldn't affect it, right?