AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage

Re:AWS switch from gzip to zstd – about 30% reduction in com... by notimetorelax on Saturday August 20, 2022 @05:02AM (#32530547)

This was true a few years back, nowadays it’s cheaper and faster to compress the data at rest as the bottleneck is frequently IO and storage space. Both, in terms of capacity and cost.

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by danieldk on Saturday August 20, 2022 @06:37AM (#32530928)

This was true a few years back

Only temporarily with SSDs. With spinning rust, it also often paid off to compress data. We'd store large treebanks compressed, because decompression was much faster than disk reads.

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by LinAGKar on Saturday August 20, 2022 @05:11AM (#32530600)

It would still produce some CPU overhead, and thus some energy usage.

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by IntelMiner on Saturday August 20, 2022 @05:34AM (#32530696)

Presumably it's the tradeoff of CPU overhead versus disk and bandwidth (larger files take longer to copy into memory, which is also energy usage. And more bandwidth to shunt around Amazon's own network)

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by SuperQue on Saturday August 20, 2022 @09:03AM (#32531771)

There's also a latency component.

Since CPUs are fast enough to deflate in real-time now, your bottleneck for a read is your storage/network.

Reducing the bytes read from storage improves the IO latency.

[ Parent | on HN ]