AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage

Re:AWS switch from gzip to zstd – about 30% reduction in com... by infogulch on Saturday August 20, 2022 @03:04AM (#32529896)

I believe zstd can detect incompressible data quickly and falls back to a low-overhead passthrough mode in that case. So you can just run everything through zstd and it handles those cases pretty optimally already.

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by le-mark on Saturday August 20, 2022 @01:31PM (#32533730)

Hmm this made me chuckle, is that not equivalent to a test for randomness, which is very very hard indeed?

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by hexxagone on Saturday August 20, 2022 @04:35PM (#32534978)

It is not. Just look at the frequencies in the data (histogram) and compute the order 0 entropy. Then you know if your data is compressible.

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by Cyberdog on Saturday August 20, 2022 @02:05PM (#32533978)

I think a useful test for this case could be pretty simple. Compress maybe 10% of the file (up to some reasonable maximum, like maybe a few megabytes) and check the compression ratio. If it's not something better than maybe 1.1x, give up, throw out the compressed data, and just pass through the file. Otherwise keep going.

[ Parent | on HN ]