News for hackers. Stuff that matters.
Re:Improving compression at scale with Zstandard by m0zg on Wednesday December 19, 2018 @05:11PM (#18720262)
I hope they pay greater attention to the low and high end of their compression ratio spectrum. On the low end, it'd be great if it could exceed lz4 in terms of speed and memory savings. On the high end it'd be great to exceed XZ/LZMA.

Right now it's impressive "in the middle", but I find myself in a lot of situations where I care about the extremes. I.e. for something that will be transferred a lot, or cold-stored, I want maximum compression, CPU/RAM usage be damned, within reason. So I tend to use LZMA there if files aren't too large. For realtime/network RPC scenarios I want minimum RAM/CPU usage and Pareto-optimality on multi-GbE networks. This is where I use LZ4 (and used to use Snappy/Zippy).

At their scale, though, FB is surely saving many millions of dollars thanks to deploying this, both in human/machine time savings and storage savings.

Re:Improving compression at scale with Zstandard by praseodym on Wednesday December 19, 2018 @05:19PM (#18720350)
It does depend on your dataset. In my experiments, Zstandard did outperform XZ/LZMA in terms of compression ratio, but not by much. However, the massive improvement in decompression speed is what won me over.
Re:Improving compression at scale with Zstandard by terrelln on Wednesday December 19, 2018 @05:18PM (#18720337)
We recently added negative compression levels that extends the fast end of the spectrum significantly. We are also working on incrementally improving the strong end of the compression ratio spectrum. We don't expect plain zstd to compress stronger than xz, but we hope to close the gap some.
Re:Improving compression at scale with Zstandard by m0zg on Wednesday December 19, 2018 @05:29PM (#18720463)
Ah, I didn't know, thanks! I'll have to re-test.
Re:Improving compression at scale with Zstandard by JyrkiAlakuijala on Wednesday December 19, 2018 @06:44PM (#18721119)
With zstd you end up on average 5-6 % worse than LZMA in density on a variety of large test corpora, but decodes 8x faster. Brotli with --large_window 30 can get within 0.6 % of LZMA, and decodes 5x faster than LZMA.

https://encode.ru/threads/2947-large-window-brotli-results-a...