Re:AWS switch from gzip to zstd – about 30% reduction in com...
by klauspost on Monday August 22, 2022 @06:25AM
(#32549540)
You only need a single part to calculate a specific offset, assuming you have part sizes stored in metadata already (a good idea).
Each part can be max 5GiB as per S3 spec. 5120 * 4 = 20KiB.
Even if you unpack to 8*2 bytes in memory when decoding, you are still not talking a huge amount of memory.
The on-disk space is ~0.0004% as blibble calculated, and should easily be offset by the compression achieved. In MinIO we don't store indexes for files < 8MiB, so for small files there is no overhead.
If the added metadata is a problem for whatever system you are looking at, then that is a characteristic of that system and not a general problem.
Each part can be max 5GiB as per S3 spec. 5120 * 4 = 20KiB.
Even if you unpack to 8*2 bytes in memory when decoding, you are still not talking a huge amount of memory.
The on-disk space is ~0.0004% as blibble calculated, and should easily be offset by the compression achieved. In MinIO we don't store indexes for files < 8MiB, so for small files there is no overhead.
If the added metadata is a problem for whatever system you are looking at, then that is a characteristic of that system and not a general problem.