AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage

Re:AWS switch from gzip to zstd – about 30% reduction in com... by flaviut on Saturday August 20, 2022 @03:09PM (#32534428)

I'd guess by using backpressure.

Modify the compression level to try and keep the output buffer at 60% full.

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by natmaka on Sunday August 21, 2022 @06:17AM (#32538809)

It reacts to the input buffer state (in bad I/O conditions it starves).

On Linux PSI ( /proc/pressure/io ) probably provides more accurate information (the code already uses /proc/cpuinfo ).

Detail: in the fileio.c module there are lines such as:

if (oldIPos == inBuff.pos) inputBlocked++; /* input buffer is full and can't take any more : input speed is faster than consumption rate /

if ( (inputBlocked > inputPresented / 8) / input is waiting often, because input buffers is full : compression or output too slow */

This impacts a 'speedChange' variable.

Its potential values (an enum) are 'noChange', 'slower', and 'faster'. They are processed rather simply: if (speedChange == slower) { ((...)) compressionLevel ++;

if (speedChange == faster) { ((...)) compressionLevel --;

[ Parent | on HN ]