Re:AWS switch from gzip to zstd – about 30% reduction in com...
by jeffffff on Sunday August 21, 2022 @03:01PM
(#32543318)
if it's not in ram, you have to do an extra IO to look it up. i don't think you understand how precious metadata space is in a large scale storage system. if you pollute the metadata cache with useless junk like this, you can't cache as many things, your hit rate goes down, and you have to do more IO operations to service each request on average. name one popular distributed file system or object store that compresses everything by default like you are claiming. you won't be able to, because none of them do it, because it's better to leave it to the application.
Re:AWS switch from gzip to zstd – about 30% reduction in com...
by blibble on Sunday August 21, 2022 @03:16PM
(#32543451)
> if it's not in ram, you have to do an extra IO to look it up
as has been explained to you several times, you don't
> if you pollute the metadata cache with useless junk like this
the overhead is 0.0004% with 1 index entry per megabyte, and if that's too much that can be reduced by 10/100/1000/10000x that by changing the size
as we're clearly now going around in circles, I won't be responding again.
and you don't understand the algorithm if you think you need to keep the index in RAM, because you don't