Re:Improving compression at scale with Zstandard
by praseodym on Wednesday December 19, 2018 @05:19PM
(#18720350)
It does depend on your dataset. In my experiments, Zstandard did outperform XZ/LZMA in terms of compression ratio, but not by much. However, the massive improvement in decompression speed is what won me over.
Re:Improving compression at scale with Zstandard
by terrelln on Wednesday December 19, 2018 @05:18PM
(#18720337)
We recently added negative compression levels that extends the fast end of the spectrum significantly. We are also working on incrementally improving the strong end of the compression ratio spectrum. We don't expect plain zstd to compress stronger than xz, but we hope to close the gap some.
Re:Improving compression at scale with Zstandard
by m0zg on Wednesday December 19, 2018 @05:29PM
(#18720463)
Ah, I didn't know, thanks! I'll have to re-test.
Re:Improving compression at scale with Zstandard
by JyrkiAlakuijala on Wednesday December 19, 2018 @06:44PM
(#18721119)
With zstd you end up on average 5-6 % worse than LZMA in density on a variety of large test corpora, but decodes 8x faster. Brotli with --large_window 30 can get within 0.6 % of LZMA, and decodes 5x faster than LZMA.
https://encode.ru/threads/2947-large-window-brotli-results-a...
Right now it's impressive "in the middle", but I find myself in a lot of situations where I care about the extremes. I.e. for something that will be transferred a lot, or cold-stored, I want maximum compression, CPU/RAM usage be damned, within reason. So I tend to use LZMA there if files aren't too large. For realtime/network RPC scenarios I want minimum RAM/CPU usage and Pareto-optimality on multi-GbE networks. This is where I use LZ4 (and used to use Snappy/Zippy).
At their scale, though, FB is surely saving many millions of dollars thanks to deploying this, both in human/machine time savings and storage savings.