Re:Modern LZ Compression (2019)
by chriswarbo on Tuesday August 17, 2021 @08:51PM
(#28216547)
> For our goal of a high compression LZ variant, we will want to Huffman code our symbols (zstd actually uses FSE, a variant of arithmetic coding to do this, but we will cover that in a future post).
[ Parent
| on HN ]
Re:Modern LZ Compression (2019)
by user-the-name on Tuesday August 17, 2021 @08:57PM
(#28216583)
But why? Why spend that much effort introducing a method that is basically entirely outdated at this point, and is also fairly complicated?
Huffmann is rather simple and hard to beat in decompression speed.
For high entropy data (=somewhat random data), FSE is quite comparable to Huffman in compression speed.
For low entropy (lots of high probability symbols, like zeroes), Huffman is about 2-3x faster. But on the flipside, FSE achieves markedly higher compression ratio.
I think FSE is worth the speed tradeoff vs Huffman in most cases.
"FSE achieves markedly higher compression ratio". I do not thin it is true, FSE/ANS achieves slightly better ratios in general.
Zstd uses both Huffman (for large alphabets) and FSE (for small alphabets).
Re:Modern LZ Compression (2019)
by user-the-name on Wednesday August 18, 2021 @06:15AM
(#28219363)
Arithmetic coding is much, much simpler. And decompression speed is not a limiting factor in most applications of data compression like this.
"Arithmetic coding is much, much simpler." Let us agree to disagree.
"And decompression speed is not a limiting factor in most applications of data compression like this". It depends on the application. Zstd and Brotli are certainly aiming at the fastest decompression speed possible.
Re:Modern LZ Compression (2019)
by user-the-name on Wednesday August 18, 2021 @02:07PM
(#28224942)
Arithmetic coding can be implemented in as little as maybe ten lines of code. It is far simpler than Huffman coding.