News for hackers. Stuff that matters.
Re:Massive Speed Gains via Parallelized BZIP2 Compression by wmf on Wednesday May 30, 2012 @02:54PM (#4044244)
BTW, bz2 is kinda over. Check out xz and the parallel version pxz.
Re:Massive Speed Gains via Parallelized BZIP2 Compression by vasi on Wednesday May 30, 2012 @03:00PM (#4044274)
I've also got my version of parallel xz: https://github.com/vasi/pixz

It doesn't require use of large temporary files like pxz, and the xz files it produces can also be decompressed in parallel.

Re:Massive Speed Gains via Parallelized BZIP2 Compression by ak217 on Wednesday May 30, 2012 @04:10PM (#4044597)
vasi, thanks very much for writing pixz, I've had a great experience working with it and it sped up my code very significantly.

The only thing I wish for is more documentation and inclusion in xz-utils (which I know is not up to you, but I'm hopeful).

Re:Massive Speed Gains via Parallelized BZIP2 Compression by vasi on Wednesday May 30, 2012 @09:08PM (#4045771)
Glad to hear it's been of use :)

The author of xz is working on a parallel compression implementation of his own, which will hopefully be in future versions of xz.

Re:Massive Speed Gains via Parallelized BZIP2 Compression by LogicX on Wednesday May 30, 2012 @03:45PM (#4044486)
Looks worth trying out --

Can I suggest adding installation packages for: brew ubuntu (at least through launchpad)

Re:Massive Speed Gains via Parallelized BZIP2 Compression by SnowLprd on Wednesday May 30, 2012 @08:14PM (#4045588)
That looks very promising. As others have pointed out here, pixz would be much more accessible if one could simply install it via "brew install pixz" or "aptitude install pixz".
Re:Massive Speed Gains via Parallelized BZIP2 Compression by vasi on Wednesday May 30, 2012 @09:08PM (#4045777)
I welcome anybody who'd like to package it! I think it's already available in Arch Linux.
Re:Massive Speed Gains via Parallelized BZIP2 Compression by bacr on Wednesday May 30, 2012 @03:35PM (#4044445)
I'm no expert in compression, so its no surprise that I've never heard of xz. What are it's strengths compared to gzip, or b2z? Efficiency? Convention?
Re:Massive Speed Gains via Parallelized BZIP2 Compression by keeperofdakeys on Thursday May 31, 2012 @01:07AM (#4046489)
You've probably heard of LZMA, xz is merely an implementation of this (7zip and rar also use LZMA). The most notable differences to bzip2 are significantly longer compression time and rather moderate decompression speed increases. This makes xz quite good for packaging data for distribution, although it takes a lot longer to compress, it decompresses a lot faster (unlike bzip2) and saves a significant amount of space.
Re:Massive Speed Gains via Parallelized BZIP2 Compression by shin_lao on Wednesday May 30, 2012 @04:36PM (#4044706)
You probably heard of 7Zip and LZMA. xz uses the LZMA2 algorithm to achieve better compression ratios.
Re:Massive Speed Gains via Parallelized BZIP2 Compression by SnowLprd on Wednesday May 30, 2012 @08:12PM (#4045580)
I disagree with your assessment. I'm aware of xz and use it when file size is paramount -- which, by the way, is almost never. Most of the time, I'm willing to give up a few megabytes in order to save time, and in my tests pbzip2 crushed xz by a factor of ten when it comes to speed.

Why not use parallelized versions of xz such as pxz or pixz? Because pre-built pxz/pixz packages are nearly non-existent. When that changes, I'll consider switching formats.

Re:Massive Speed Gains via Parallelized BZIP2 Compression by vasi on Wednesday May 30, 2012 @09:07PM (#4045765)
Remember that compressors, including xz, support multiple compression levels. The default level for xz is 6, which is perhaps too far on the small-but-slow side. Levels 2 and lower tend to give similar compression levels to bzip2, and are considerably faster.

Also, note that decompressing bzip2 is very slow, xz usually beats it by a factor of two or more.

Re:Massive Speed Gains via Parallelized BZIP2 Compression by SnowLprd on Wednesday May 30, 2012 @10:22PM (#4045976)
I agree that the default level (6) for xz probably errs too much on favoring file size over speed. My tests with compression levels 1-2 do indeed show modestly improved size and speed performance relative to single-threaded bzip2.

The fact remains, however, that I can't seem to find a simple way to install a parallelized version of xz. Perhaps I'll post an issue in the Github issue tracker for pixz and see if we can't resolve that. :)

Re:Massive Speed Gains via Parallelized BZIP2 Compression by james4k on Wednesday May 30, 2012 @04:49PM (#4044755)
Recently when compressing some SQL backups, I found xz to be very slow when compared to bz2 (at least with default compression settings).

Edit: And in this case, bz2 even gave better compression ratios if I recall correctly.

Re:Massive Speed Gains via Parallelized BZIP2 Compression by ralph on Wednesday May 30, 2012 @06:48PM (#4045288)
If speed matters more than compression then consider lzop(1).