News for hackers. Stuff that matters.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by dylan604 on Saturday August 20, 2022 @01:13PM (#32533605)
How clear cut is it when I'm storing a bunch of compressed video files? It's totally a waste at that point to even attempt to compress these files.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by lazide on Saturday August 20, 2022 @03:31PM (#32534587)
What I've personally implemented is trial compression with heuristics (you eagerly compress chunks, and if enough chunks don't compress, stop trying). It does require low level input/output control and per-chunk/block compression.

That said, a surprising number of video sources use sparse file type setups, and I've gotten pretty good compression (up to 60%) using LZ4 with NVR files from some brands.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by treffer on Monday August 22, 2022 @10:12AM (#32551635)
This is not how such storage systems work. If the whole storage system is for you and you only store compressed video files on it then perhaps.

But we are talking about multi-tenant systems here. You will get better performance and/or better prices if the system finds compressible data for other customers.

All the benefits hold, even for you with non compressible data, if there is an overall benefit.

Hitting a bottleneck less often then this will reduce your tail latencies, too.

It is just that _your files_ won't contribute to these improvements. But you get all the benefits as well.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by goodpoint on Saturday August 20, 2022 @02:42PM (#32534249)
No, most compressed storage systems do not waste any CPU on data that is not compressible.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by treffer on Sunday August 21, 2022 @07:47AM (#32539246)
Having a big CPU (which you need to get many PCIe lanes) and then not using it is waste.

It would be waste to _not_ use the CPU time.

Granted, the CPU powet consumption would drop with enabled power management, but that's only marginal gains as the CPU is likely busy anyway and the total duration of busy time might drop (race to sleep)

I have used this inverted logic to great extends. E.g. when expanding data center capacity I was usually swapping old harware for new hardware once it arrived. It is way better to have the older generations as spares and use the new capacity at the most utilized places.

Do not waste the things you pay for!

Re:AWS switch from gzip to zstd – about 30% reduction in com... by goodpoint on Monday August 22, 2022 @11:18AM (#32552671)
That's not the topic of the thread.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by treffer on Monday August 22, 2022 @12:29PM (#32553722)
Ah, I thought you meant that it is a waste to do it, but yeah, most of the time a heuristic makes sure you don't waste anything
Re:AWS switch from gzip to zstd – about 30% reduction in com... by miohtama on Saturday August 20, 2022 @01:25PM (#32533693)
Video files have already entropy coding applied to them and thus any compression gains with reapplying a generic entropy coding like zstd are unlikely:

https://en.wikipedia.org/wiki/Entropy_coding

Re:AWS switch from gzip to zstd – about 30% reduction in com... by sicp-enjoyer on Saturday August 20, 2022 @02:11PM (#32534024)
That's exactly what the poster is saying.