News for hackers. Stuff that matters.

A comment on AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage. Parent comment

Re:AWS switch from gzip to zstd – about 30% reduction in com... by blibble on Saturday August 20, 2022 @07:44AM (#32531265)
> They allow you to slice an arbitrary byte range out of an object which is technically harder to implement on a compressed file.

this is pretty easy, you flush the compression buffer every megabyte or so and maintain an index

maybe 50 lines of code

[ Parent | on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com... by jeffffff on Saturday August 20, 2022 @09:08AM (#32531799)
Sure, but now you've added an extra layer of indirection which can have a significant impact on performance
[ Parent | on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com... by klauspost on Saturday August 20, 2022 @09:35AM (#32531961)
It doesn't really have to impact performance. The index is generated easily as a side-effect of compression. And the index is only needed if you need to seek.

I implemented this as part of the MinIO server. See "Seeking Compressed Files" here: https://blog.min.io/transparent-data-compression/

We choose a compressor without literal compression for a faster baseline, but the concept remains the same.

[ Parent | on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com... by jeffffff on Saturday August 20, 2022 @04:30PM (#32534955)
But if you do need to seek, which is really common in data warehouse workloads for example, unless you keep the index in ram you have to do an extra IO on every seek to read the index
[ Parent | on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com... by blibble on Saturday August 20, 2022 @05:07PM (#32535139)
there's always going to be some metadata for the file that needs to be looked up before you can start seeking (ACLs, sector/extent/cluster location, etc)

the index goes in there, no extra seek needed

[ Parent | on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com... by jeffffff on Sunday August 21, 2022 @11:01AM (#32540844)
yeah that isn't free either, it adds significant bloat to your metadata. with most enterprise customers encrypting and/or compressing data before putting it into s3, it doesn't seem like there would be much benefit. s3 really isn't the right layer to implement compression. filesystems aren't either. it's better to leave it up to the application.
[ Parent | on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com... by blibble on Sunday August 21, 2022 @11:30AM (#32541129)
> yeah that isn't free either, it adds significant bloat to your metadata

yeah, 4 bytes for every megabyte

> s3 really isn't the right layer to implement compression. filesystems aren't either. it's better to leave it up to the application.

yeah, I'm sure you're right and Amazon have absolutely no idea what they're doing and like to spend unnecessary CPU cycles doing pointless work and add "significant bloat" to their metadata

... or, you're wrong (like in every previous comment in this chain)

[ Parent | on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com... by jeffffff on Sunday August 21, 2022 @12:10PM (#32541534)
https://www.reddit.com/r/programming/comments/wtd61q/aws_swi...

this tweet is not talking about compressing customer data in s3, i seriously doubt that aws compresses customer data in s3 for all the reasons i've already listed. i am right and amazon does know what they're doing, which is why they don't compress customer data in s3.

4 bytes per megabyte becomes significant at scale when you have to keep it in ram, which you have to do if you want to avoid the extra IO.

[ Parent | on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com... by klauspost on Monday August 22, 2022 @06:25AM (#32549540)
You only need a single part to calculate a specific offset, assuming you have part sizes stored in metadata already (a good idea).

Each part can be max 5GiB as per S3 spec. 5120 * 4 = 20KiB.

Even if you unpack to 8*2 bytes in memory when decoding, you are still not talking a huge amount of memory.

The on-disk space is ~0.0004% as blibble calculated, and should easily be offset by the compression achieved. In MinIO we don't store indexes for files < 8MiB, so for small files there is no overhead.

If the added metadata is a problem for whatever system you are looking at, then that is a characteristic of that system and not a general problem.

[ Parent | on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com... by blibble on Sunday August 21, 2022 @12:59PM (#32542066)
ah yes, "authoritative" comments from random reddit accounts

and you don't understand the algorithm if you think you need to keep the index in RAM, because you don't

[ Parent | on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com... by jeffffff on Sunday August 21, 2022 @03:01PM (#32543318)
if it's not in ram, you have to do an extra IO to look it up. i don't think you understand how precious metadata space is in a large scale storage system. if you pollute the metadata cache with useless junk like this, you can't cache as many things, your hit rate goes down, and you have to do more IO operations to service each request on average. name one popular distributed file system or object store that compresses everything by default like you are claiming. you won't be able to, because none of them do it, because it's better to leave it to the application.
[ Parent | on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com... by blibble on Sunday August 21, 2022 @03:16PM (#32543451)
> if it's not in ram, you have to do an extra IO to look it up

as has been explained to you several times, you don't

> if you pollute the metadata cache with useless junk like this

the overhead is 0.0004% with 1 index entry per megabyte, and if that's too much that can be reduced by 10/100/1000/10000x that by changing the size

as we're clearly now going around in circles, I won't be responding again.

[ Parent | on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com... by jeffffff on Sunday August 21, 2022 @10:43PM (#32546723)
well i tried. i wish you the best of luck in your continued ignorance.
[ Parent | on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com... by blibble on Monday August 22, 2022 @04:08AM (#32548637)
the irony
[ Parent | on HN ]