News for hackers. Stuff that matters.
Re:Zstandard – Fast real-time compression algorithm by waterhouse on Monday November 26, 2018 @01:58AM (#18531628)
Standard comment on zstd: The "zstdcat" program (equivalent to "zstd -cdfq", I think) is capable of reading a few other compression formats, including .gz and .lz4 (apparently .lzma and .xz are available too), if support is built in at compile time. Unbelievably, zstdcat is 1.5x faster (in my experience) at decoding .gz files than zcat / gunzip / gzip -d. I dare you to try it.

Anyway, this should ease transitions away from legacy compression formats.

Re:Zstandard – Fast real-time compression algorithm by stochastic_monk on Monday November 26, 2018 @02:53AM (#18531784)
It’s also quite convenient supporting a variety of compressed formats with one API.
Re:Zstandard – Fast real-time compression algorithm by goodside on Monday November 26, 2018 @09:31AM (#18533562)
If you find 1.5x-faster gzip useful, you should consider benchmarking pigz, a parallel gzip implementation that goes faster still: https://zlib.net/pigz/

Similarly there's lbzip2 ( http://lbzip2.org ) for parallel bz2.

Re:Zstandard – Fast real-time compression algorithm by waterhouse on Monday November 26, 2018 @11:38AM (#18534545)
Good point. But for my use case, I have hundreds or more of logfiles, and I parallelize at that level already. So I'm concerned with the total CPU time per file more than anything else.