Re:Ebiggers/libdeflate: Heavily optimized DEFLATE/zlib/gzip ...
by powturbo on Saturday September 02, 2023 @01:39AM
(#37358817)
Thank for your analyse. It seems it's a corner case, benchmarking a base64 encoded random file is not a typical case.
igzip has no more advantage above libdeflate. It has only fast compression at the levels 0,1,2 but with mediocre compression ratio.
I have already incorporated the ISA-L Huffman decoder into rapidgzip but that did not give full ISA-L speed. Ergo, I think the last performance comes from the inflate loop (decode Huffman, decode distance code if necessary, resolve references, repeat). This part is written in Assembler and seems to do some kind of speculative prefetching, i.e., already get the next Huffman Code symbol assuming that the current symbol is a literal or something like that. It's quite interesting but I doubt, or rather, I already tried a bit and failed to reproduce this kind of prefetching inside the C++ code but failed to do so. The compiler is probably rearranging everything anyway.
rapidgzip --analyze 4GiB-base64.gz | tail -100