News for hackers. Stuff that matters.
Re:Minimalist Guide to Lossless Compression (2019) by magicalhippo on Monday December 13, 2021 @05:18AM (#29538311)
> zstd at compression level 1 can do ~2GB/s per core

So unless you can multithread that workload, it's already behind by a factor of 2.

> Computers are plenty fast.

My point was you can no longer assume the disk is significantly slower, at least for streaming workloads. You can often still win by spending CPU cycles doing clever stuff, but it's not several orders of magnitude difference like it used to be.

Re:Minimalist Guide to Lossless Compression (2019) by willis936 on Monday December 13, 2021 @06:09AM (#29538553)
Most zstd implementations are multithreaded for single compression and decompression tasks. My home server has 24 cores and is using 16x PCIe 3.0 for storage (16 GB/s). Through benchmarking I found that I am storage bound from zstd-3+ and compute bound from zstd-2 and zstd-1. I run zstd-3.
Re:Minimalist Guide to Lossless Compression (2019) by 1_player on Monday December 13, 2021 @06:23AM (#29538609)
Dumb question, but shouldn't it be the opposite? The lower the compression level, the faster the CPU can process the data, making storage the bottleneck. Higher compression level -> CPU becomes the bottleneck.
Re:Minimalist Guide to Lossless Compression (2019) by willis936 on Monday December 13, 2021 @06:53AM (#29538768)
I believe in the TrueNAS interface I use these are negative compression levels. I could be wrong. It was just some empirical observations. I may flesh out my matrix more to paint a better picture.