W.r.t. to ZSTD: The usual thing is to provide a zlib compatible API, which ZSTD does ( https://github.com/facebook/zstd/tree/dev/zlibWrapper ). But the ZSTD zlib compatibility layer causes lower performance, so it is better to use it directly. Maybe all future compression algorithms can provide a ZSTD compatible API, so we have to do less work in the future?
Could it? That doesn't sound very portable, and ZFS works on FreeBSD too.
Frequently Asked Questions
“Q: Why is it so many lines of code, I can't review all of that...
A: Most of this code is the ZSTD library, which has not been altered.”
One reason this needs testing because this is a file system. If it breaks, it can lose you much more data than the file being worked on.
There also may be serious performance degradation on hardware or configurations the developers didn’t look at.
There also is some new code added to call the zstd library.
Because filesystems need to store compression metadata and compressor settings differently than individual archive streams do. Because of the way ZFS stores configuration like this, in the previous version of these patches, they had to choose a subset of the available compression levels when adding zstd support.
Different compressors and decompressors also have different state sizes for different settings. Allocating, reusing, and discarding buffers for compression/decompression state in a sensible way inside an operating system kernel is not trivial.