Re:Improving compression at scale with Zstandard
by JyrkiAlakuijala on Wednesday December 19, 2018 @06:40PM
(#18721090)
zstd vs. brotli -- there are loooots of badly conducted experiments. Even the zstd folks themselves fell into the trap of comparing brotli and zstd with different window sizes.
I never was able to locate a file that compresses better with zstd than with brotli, if you use the same window size in both.
The car analogy in using different window sizes is to drive on 1st gear with one car and drive on 5th gear with other, i.e., meaningless comparisons. One needs to use the same window size with both algos to be able to understand the performance potential in them. When one does that, brotli always wins in density and most of the time on compression speed to a given density. Zstd wins in decompression speed, but neither of them is slow (think 600 MB/s vs. 900 MB/s).
https://github.com/google/brotli/issues/642 is relatively well conducted, but covers the slow+dense and large window compression only.
Unfortunately, we have a strict requirement that the decompression be done in pure Java (and ideally providing an InputStream interface). The official Brotli repo includes a pure Java decompression library that took about three lines of code to drop in next to our gzip support, while zstd had nothing suitable (a JNI wrapper is not an option, and the Java decompressor at [1] does not provide an streaming interface). Writing my own Java decompressor for zstd sounded like a fun project, but that would have been too much like yak shaving; brotli is good enough.
[1] https://github.com/airlift/aircompressor/tree/master/src/mai...