Using machine learning to choose compression algorithms

Re:Using machine learning to choose compression algorithms by mappu on Sunday December 08, 2019 @06:19PM (#21739160)

Are all compressors simply being run with their default arguments? There's a lot of scope for speed/filesize tradeoffs within a single compressor.

EDIT: You are missing `csv+zstd` ? It should obsolete `csv+gzip` at all speeds and compression levels.

There is a pareto-optimality frontier here - I ran my testing back in 2016 https://code.ivysaur.me/compression-performance-test/ but the numbers are now a little bit obsolete (e.g. zstd and brotli have both seen a lot of improvements).

[ Parent | on HN ]

Re:Using machine learning to choose compression algorithms by kootenpv on Monday December 09, 2019 @04:34AM (#21741387)

Yea, I thought parameters per compression algorithm should indeed be added in a next version :) more compute but definitely an improvement. I think pandas doesn't offer zstd as option with csv, but I'll check once more.

EDIT: indeed, it's missing in `to_csv` - seems like an oversight.

[ Parent | on HN ]

Re:Using machine learning to choose compression algorithms by alexott on Monday December 09, 2019 @01:09AM (#21740733)

+1 - and if you could use zstd with custom dictionary, then you can achieve even better compression ratios

[ Parent | on HN ]

Re:Using machine learning to choose compression algorithms by kootenpv on Monday December 09, 2019 @04:32AM (#21741383)

Yea zstd is really amazing... if I would choose a single one all the time it'd be zstd for sure.

[ Parent | on HN ]