Re:Data Compression Drives the Internet. Here’s How It Works
by magicalhippo on Saturday June 10, 2023 @06:03PM
(#36275632)
OpenZFS' integration of Zstandard uses LZ4 as a "compression canary" for higher ZStandard compression levels, where they feed the data blocks through LZ4 and if it compresses it enough, feeds it through Zstandard.
This relies on LZ4 being very fast, especially with it's early-exit on incompressible data.
Overall this turns out to be a win, you lose a little bit of compression at a huge decrease in CPU over just using the same Zstandard compression for all the blocks.
This relies on LZ4 being very fast, especially with it's early-exit on incompressible data.
Overall this turns out to be a win, you lose a little bit of compression at a huge decrease in CPU over just using the same Zstandard compression for all the blocks.