News for hackers. Stuff that matters.

A comment on Intel QuickAssist Technology Zstandard Plugin for Zstandard. Parent comment

Re:Intel QuickAssist Technology Zstandard Plugin for Zstandard by wolf550e on Thursday August 17, 2023 @05:08AM (#37159071)
It's obsolete. DEFLATE format is limited to 32KB LZ window with huffman coding.

Zstd can use a much larger window (8MB recommended) and a much better entropy coder: https://github.com/Cyan4973/FiniteStateEntropy

[ Parent | on HN ]
Re:Intel QuickAssist Technology Zstandard Plugin for Zstandard by klauspost on Thursday August 17, 2023 @12:14PM (#37163963)
Last I checked QAT was limited to 64KB backreferences and independent 128KB blocks.

Now you know why they are comparing 16KB payloads only.

[ Parent | on HN ]
Re:Intel QuickAssist Technology Zstandard Plugin for Zstandard by terrelln on Thursday August 17, 2023 @12:54PM (#37164634)
Yeah, that is definitely a limitation with QAT. It isn't a great fit for larger data, as it can quickly lose compression ratio due to its smaller window. However, there is a lot of compression done on data that is <64KB. E.g. compression in RocksDB.

We also have some half baked ideas to combine a fast SW match finder that only looks for matches >64KB away, and supplements the matches that QAT finds.

[ Parent | on HN ]