News for hackers. Stuff that matters.
Re:Encrypted Backup Shootout by daneel_w on Saturday January 02, 2021 @10:47PM (#25619216)
gzip is a particularly poor performer when it comes to spotting repetition across larger amounts of data. Use bzip2 instead if you want to give home-baked tarballs a somewhat fair competition against Bupstash etc.
Re:Encrypted Backup Shootout by garmaine on Sunday January 03, 2021 @12:52AM (#25619834)
Or lzop for speed, while still being better than gzip.
Re:Encrypted Backup Shootout by garaetjjte on Sunday January 03, 2021 @08:15PM (#25626952)
For speed, LZ4 is great.
Re:Encrypted Backup Shootout by rzzzt on Sunday January 03, 2021 @06:00AM (#25621050)
There's also lrzip for large files: https://github.com/ckolivas/lrzip
Re:Encrypted Backup Shootout by andrewchambers on Saturday January 02, 2021 @11:07PM (#25619338)
Thanks for the suggestion, I mainly used gzip as that is what my muscle memory goes for when compressing tarballs, it is a fair criticism.
Re:Encrypted Backup Shootout by petre on Sunday January 03, 2021 @01:27AM (#25619980)
I found out that xz -1 produces smaller files than gzip -6, faster. That's what we use now.
Re:Encrypted Backup Shootout by wolf550e on Sunday January 03, 2021 @06:51AM (#25621255)
You should try zstd, it often beats xz. xz beats zstd only in high compression scenarios (-9).
Re:Encrypted Backup Shootout by smichel17 on Saturday January 02, 2021 @11:21PM (#25619403)
How does that compare to xz?
Re:Encrypted Backup Shootout by Aachen on Saturday January 02, 2021 @11:33PM (#25619477)
In my experience it depends on the type of data, though never have I seen it do better or worse than a negligible amount (on real world data, I'm sure you could find the type it's better at and exploit that).
Re:Encrypted Backup Shootout by brnt on Saturday January 02, 2021 @11:32PM (#25619472)
Or zstd?
Re:Encrypted Backup Shootout by Aachen on Saturday January 02, 2021 @11:34PM (#25619480)
Want zstd more focused on performance without too big a decrease in compression ratio, rather than getting better compression ratios? I might be mistaken.
Re:Encrypted Backup Shootout by aidenn0 on Sunday January 03, 2021 @02:33AM (#25620247)
zstd's performance curve is entirely within gzip's. For every level of gzip compression, zstd has a level with better compression at the same speed, and a level with the same (or better) compression at a higher speed. Note that this is all for single core; zstd can also run in parallel.
Re:Encrypted Backup Shootout by Aachen on Sunday January 03, 2021 @12:02PM (#25622997)
That's really cool, thanks for the info!
Re:Encrypted Backup Shootout by daneel_w on Sunday January 03, 2021 @09:33AM (#25621973)
xz for the most part does a much better job than bzip2.
Re:Encrypted Backup Shootout by wolf550e on Sunday January 03, 2021 @06:52AM (#25621260)
Both gzip and bzip2 are very old. You should use zstd.
Re:Encrypted Backup Shootout by daneel_w on Sunday January 03, 2021 @09:33AM (#25621964)
Or xz, my current favorite.