Re:Encrypted Backup Shootout
by daneel_w on Saturday January 02, 2021 @10:47PM
(#25619216)
gzip is a particularly poor performer when it comes to spotting repetition across larger amounts of data. Use bzip2 instead if you want to give home-baked tarballs a somewhat fair competition against Bupstash etc.
[ Parent
| on HN ]
Or lzop for speed, while still being better than gzip.
Re:Encrypted Backup Shootout
by garaetjjte on Sunday January 03, 2021 @08:15PM
(#25626952)
For speed, LZ4 is great.
There's also lrzip for large files: https://github.com/ckolivas/lrzip
Re:Encrypted Backup Shootout
by andrewchambers on Saturday January 02, 2021 @11:07PM
(#25619338)
Thanks for the suggestion, I mainly used gzip as that is what my muscle memory goes for when compressing tarballs, it is a fair criticism.
I found out that xz -1 produces smaller files than gzip -6, faster. That's what we use now.
You should try zstd, it often beats xz. xz beats zstd only in high compression scenarios (-9).
How does that compare to xz?
In my experience it depends on the type of data, though never have I seen it do better or worse than a negligible amount (on real world data, I'm sure you could find the type it's better at and exploit that).
Or zstd?
Want zstd more focused on performance without too big a decrease in compression ratio, rather than getting better compression ratios? I might be mistaken.
zstd's performance curve is entirely within gzip's. For every level of gzip compression, zstd has a level with better compression at the same speed, and a level with the same (or better) compression at a higher speed. Note that this is all for single core; zstd can also run in parallel.
That's really cool, thanks for the info!
xz for the most part does a much better job than bzip2.
Both gzip and bzip2 are very old. You should use zstd.