News for hackers. Stuff that matters.
Re:Improving compression at scale with Zstandard by terrelln on Wednesday December 19, 2018 @04:47PM (#18720041)
Porting is generally very easy.

* If you already have the compression algorithm tagged, through a file extension, or a field, then you can use that to dispatch to the right decompression algorithm.

* Zlib, gzip, xz, zstd, ... all have headers. If you are using zlib, and switching to zstd, you simply have to check the first 4 bytes for the zstd header using ZSTD_isFrame() [0], or attempting to decompress with zstd and if it fails fall back to the previous decompression algorithm.

* The zstd CLI can decompress both zstd and zlib/gzip if compiled with zlib support.

* Zstd provides a wrapper around the zlib API so you could transparently switch to zstd. [1]

[0] https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 [1] https://github.com/facebook/zstd/tree/dev/zlibWrapper

Re:Improving compression at scale with Zstandard by jclay on Wednesday December 19, 2018 @04:53PM (#18720096)
Great, thanks! Last question.

On the blog you mention you are underway porting the internal code to replace Zlib with Zstd. Is there a reason you decided not to use the wrapper as a first pass to migrate all uses of Zlib to Zstd across the entire codebase?

Re:Improving compression at scale with Zstandard by terrelln on Wednesday December 19, 2018 @05:14PM (#18720290)
There are a few reasons we haven't used the wrapper.

* The larger services require tuning to get the best performance out of zstd, and we use some advanced options.

* We have a "Managed Compression" library which does zstd dictionary compression, which doesn't work with the wrapper.

* We have our own automatic decompression framework that handles many algorithms [0].

* A lot of use cases switched over from other algorithms than zlib.

* A lot of use cases switched over to zstd organically, without our involvement, since it was such a clear win.

[0] https://github.com/facebook/folly/blob/master/folly/compress...