News for hackers. Stuff that matters.
Re:Improving compression at scale with Zstandard by jclay on Wednesday December 19, 2018 @04:40PM (#18719970)
That sounds fantastic. What is the porting process generally like? Are there any possibilities to create an API compatible wrapper to make it a drop in replacement for zlib?
Re:Improving compression at scale with Zstandard by terrelln on Wednesday December 19, 2018 @04:47PM (#18720041)
Porting is generally very easy.

* If you already have the compression algorithm tagged, through a file extension, or a field, then you can use that to dispatch to the right decompression algorithm.

* Zlib, gzip, xz, zstd, ... all have headers. If you are using zlib, and switching to zstd, you simply have to check the first 4 bytes for the zstd header using ZSTD_isFrame() [0], or attempting to decompress with zstd and if it fails fall back to the previous decompression algorithm.

* The zstd CLI can decompress both zstd and zlib/gzip if compiled with zlib support.

* Zstd provides a wrapper around the zlib API so you could transparently switch to zstd. [1]

[0] https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 [1] https://github.com/facebook/zstd/tree/dev/zlibWrapper

Re:Improving compression at scale with Zstandard by jclay on Wednesday December 19, 2018 @04:53PM (#18720096)
Great, thanks! Last question.

On the blog you mention you are underway porting the internal code to replace Zlib with Zstd. Is there a reason you decided not to use the wrapper as a first pass to migrate all uses of Zlib to Zstd across the entire codebase?

Re:Improving compression at scale with Zstandard by terrelln on Wednesday December 19, 2018 @05:14PM (#18720290)
There are a few reasons we haven't used the wrapper.

* The larger services require tuning to get the best performance out of zstd, and we use some advanced options.

* We have a "Managed Compression" library which does zstd dictionary compression, which doesn't work with the wrapper.

* We have our own automatic decompression framework that handles many algorithms [0].

* A lot of use cases switched over from other algorithms than zlib.

* A lot of use cases switched over to zstd organically, without our involvement, since it was such a clear win.

[0] https://github.com/facebook/folly/blob/master/folly/compress...

Re:Improving compression at scale with Zstandard by megous on Wednesday December 19, 2018 @04:46PM (#18720037)
Rather simple. For example here's my port of qemu to use zstd compression algorithm for QCOW2 images, instead of zlib:

https://megous.com/git/qemu-zstd/commit/?id=45df9c0510e737e6...

Re:Improving compression at scale with Zstandard by kstrauser on Wednesday December 19, 2018 @07:43PM (#18721490)
Awesome! What kind of metrics differences have you seen from the change?
Re:Improving compression at scale with Zstandard by megous on Wednesday December 19, 2018 @08:38PM (#18721806)
Much faster and/or better compression/decompression of images (tunable of course, depending on what level you set in the code), and also faster startup of VMs. I haven't measured exactly. I'd guess ~30% in my case.

But all this depends on where the bottleneck is in any particular case. My VMs are on HDD, so increased decompression speed doesn't matter that much, but reduced size helps reading the necessary data faster. Linux VMs seem to be more compressible than Windows ones.

It's certainly better than zlib in any case.

Re:Improving compression at scale with Zstandard by Cyan4973 on Wednesday December 19, 2018 @04:44PM (#18720013)
There is a zlib wrapper included in the project : https://github.com/facebook/zstd/tree/master/zlibWrapper