News for hackers. Stuff that matters.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by treffer on Saturday August 20, 2022 @07:30AM (#32531190)
I think it is a clear cut, mostly because I do not think that compression compromises any of those features, all while making the user experience better.

For any storage system like this you usually have a few bottlenecks. IO and Network are the obvious ones, followed by tiering (cache, fast io, slow io, ...) and at the very end CPU.

Now let's say network is your bottleneck. If you can send the data to the client in a compressed for then you get the compression ratio as additional bandwidth. And the user would get the data quicker! So compression to the network is a clear win.

But the common bottleneck is often IO, a high end SSDs with 1M IOPS at 4KB would _theoretically_ serve 4GB/s, a 40GBit link. That's without any redundancy over other overhead.

Again compression to the storage layer would decrease the total amount of IOs, thus making sure a customer gets data quicker.

Ok, let's say both are not the issue. The fastest compression algorithms compete with memcopy. So if you need just one copy of your data you might have been faster by compressing it.

Especially fast compression algorithms (zstd, lz4, snappy, lzo, ...) are worth the CPU cost with virtually no downsides. The problem is finding the right sweet spot that reduces the current bottleneck without creating a CPU bottleneck, but zstd offers the greatest flexibility there, too.

Oh for range requests.... Those large objects are likely split anyway, for easier error recovery (imagine 100MB into a 1GB transfer you notice that the file data was corrupted - not good). Once you work on blocks it's easy to do somewhat efficient range requests again.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by dylan604 on Saturday August 20, 2022 @01:13PM (#32533605)
How clear cut is it when I'm storing a bunch of compressed video files? It's totally a waste at that point to even attempt to compress these files.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by lazide on Saturday August 20, 2022 @03:31PM (#32534587)
What I've personally implemented is trial compression with heuristics (you eagerly compress chunks, and if enough chunks don't compress, stop trying). It does require low level input/output control and per-chunk/block compression.

That said, a surprising number of video sources use sparse file type setups, and I've gotten pretty good compression (up to 60%) using LZ4 with NVR files from some brands.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by treffer on Monday August 22, 2022 @10:12AM (#32551635)
This is not how such storage systems work. If the whole storage system is for you and you only store compressed video files on it then perhaps.

But we are talking about multi-tenant systems here. You will get better performance and/or better prices if the system finds compressible data for other customers.

All the benefits hold, even for you with non compressible data, if there is an overall benefit.

Hitting a bottleneck less often then this will reduce your tail latencies, too.

It is just that _your files_ won't contribute to these improvements. But you get all the benefits as well.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by goodpoint on Saturday August 20, 2022 @02:42PM (#32534249)
No, most compressed storage systems do not waste any CPU on data that is not compressible.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by treffer on Sunday August 21, 2022 @07:47AM (#32539246)
Having a big CPU (which you need to get many PCIe lanes) and then not using it is waste.

It would be waste to _not_ use the CPU time.

Granted, the CPU powet consumption would drop with enabled power management, but that's only marginal gains as the CPU is likely busy anyway and the total duration of busy time might drop (race to sleep)

I have used this inverted logic to great extends. E.g. when expanding data center capacity I was usually swapping old harware for new hardware once it arrived. It is way better to have the older generations as spares and use the new capacity at the most utilized places.

Do not waste the things you pay for!

Re:AWS switch from gzip to zstd – about 30% reduction in com... by goodpoint on Monday August 22, 2022 @11:18AM (#32552671)
That's not the topic of the thread.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by treffer on Monday August 22, 2022 @12:29PM (#32553722)
Ah, I thought you meant that it is a waste to do it, but yeah, most of the time a heuristic makes sure you don't waste anything
Re:AWS switch from gzip to zstd – about 30% reduction in com... by miohtama on Saturday August 20, 2022 @01:25PM (#32533693)
Video files have already entropy coding applied to them and thus any compression gains with reapplying a generic entropy coding like zstd are unlikely:

https://en.wikipedia.org/wiki/Entropy_coding

Re:AWS switch from gzip to zstd – about 30% reduction in com... by sicp-enjoyer on Saturday August 20, 2022 @02:11PM (#32534024)
That's exactly what the poster is saying.