News for hackers. Stuff that matters.
Re:Introduce ZSTD compression to ZFS by d33 on Sunday May 17, 2020 @06:42AM (#23211072)
There's one thing I don't understand. Each time a new compression algorithm is introduced, it's the Next Big Thing. Why isn't the implementation of the algorithm as simple as linking in the related library, assuming they'd all have a similar interface? After all, it seems like what you need is a header and a function that converts a compressed block to a decompressed one and the other way round. Where's the complexity from the API/implementation perspective given that a library is already there?
Re:Introduce ZSTD compression to ZFS by tobias3 on Sunday May 17, 2020 @07:42AM (#23211296)
In this case it could use the Linux crypto API, which already has ZSTD support and provides compress/decompress functions. But that is exported as GPLv2 so ZFS needs to do its own thing. And idk if the API is sufficient w.r.t. to e.g. workspace management/reuse.

W.r.t. to ZSTD: The usual thing is to provide a zlib compatible API, which ZSTD does ( https://github.com/facebook/zstd/tree/dev/zlibWrapper ). But the ZSTD zlib compatibility layer causes lower performance, so it is better to use it directly. Maybe all future compression algorithms can provide a ZSTD compatible API, so we have to do less work in the future?

Re:Introduce ZSTD compression to ZFS by 1over137 on Sunday May 17, 2020 @10:50AM (#23212401)
>In this case it could use the Linux crypto API

Could it? That doesn't sound very portable, and ZFS works on FreeBSD too.

Re:Introduce ZSTD compression to ZFS by glandium on Monday May 18, 2020 @07:43AM (#23221116)
Zstd is dual licensed BSD and GPL. It is kind of weird the kernel limits it to GPL modules.
Re:Introduce ZSTD compression to ZFS by georgyo on Sunday May 17, 2020 @06:51AM (#23211106)
For one thing, kernel models don't link against libraries. So a new implementation is almost always written.
Re:Introduce ZSTD compression to ZFS by Someone on Sunday May 17, 2020 @07:10AM (#23211176)
This incorporates the zstd source code ‘as is’. FTA:

Frequently Asked Questions

“Q: Why is it so many lines of code, I can't review all of that...

A: Most of this code is the ZSTD library, which has not been altered.”

One reason this needs testing because this is a file system. If it breaks, it can lose you much more data than the file being worked on.

There also may be serious performance degradation on hardware or configurations the developers didn’t look at.

There also is some new code added to call the zstd library.

Re:Introduce ZSTD compression to ZFS by m4rtink on Sunday May 17, 2020 @10:59AM (#23212456)
It often works that way - when binary RPM compression switched from xz to ZSTD in Fedora recently, it was reaaly just about making sure the tooling can now read ZSTD while keepin xz support in place for compatibility (basically msking sure it links to ZSTD and can use it at runtime). Then just switch the Fedora build system to compress binary RPMs with ZSTD when they are built and you are done. :)
Re:Introduce ZSTD compression to ZFS by vkaku on Sunday May 17, 2020 @04:35PM (#23215211)
It's usually the same as linking, except there are tunables in every library, and the hard part is figuring out which one is appropriate for the job. Maybe, this call is to ask people how this library works out in the real world, on the current datasets that people have. Besides, they are talking of testing implementation specific limitations, and may they want to test for that.
Re:Introduce ZSTD compression to ZFS by microcolonel on Sunday May 17, 2020 @03:42PM (#23214720)
> Why isn't the implementation of the algorithm as simple as linking in the related library

Because filesystems need to store compression metadata and compressor settings differently than individual archive streams do. Because of the way ZFS stores configuration like this, in the previous version of these patches, they had to choose a subset of the available compression levels when adding zstd support.

Different compressors and decompressors also have different state sizes for different settings. Allocating, reusing, and discarding buffers for compression/decompression state in a sensible way inside an operating system kernel is not trivial.