News for hackers. Stuff that matters.
Re:Xz by gxti on Thursday June 24, 2010 @04:33PM (#1459034)
I've found that on many types of data, xz -2 is twice as fast and compresses better than gzip -9. Of course gzip -1 kicks the pants off xz but there's absolutely no reason to use gzip at its higher levels, nor to use bzip2 at all. And if you want fast, LZO does it better -- the lzop tool provides a typical unix compress interface.

Using 50MiB from a tarball I had lying around, best of 3:

     prog    time(s)   size(%)
    gzip  -1  1.703    38.7
    gzip  -9 14.670    35.1    (worse than xz -1)
    bzip2 -1  7.714    34.0
    bzip2 -9  8.035    31.4    (worse than xz -1)
    xz    -1  6.278    31.3
    xz    -9 42.445    22.7    (best compression)
    lzop  -1  0.670    46.1    (fastest)
    lzop  -9 18.144    38.0
Re:Xz by gxti on Friday June 25, 2010 @09:58AM (#1460733)
If anyone is still reading this thread, I made a chart overnight using a better corpus:

http://partiallystapled.com/~gxti/misc/2010/06/25-shootout.p...

http://partiallystapled.com/~gxti/misc/2010/06/25-linux-2.6....

Re:Xz by huhtenberg on Thursday June 24, 2010 @05:34PM (#1459205)
Out of curiousity, can you run this tarball through paq (say, paq8hp12)?

http://en.wikipedia.org/wiki/PAQ

Re:Xz by gxti on Thursday June 24, 2010 @06:39PM (#1459374)
It segfaulted at the end, after taking 7m28s. I don't know how big the result was because it got truncated to 35 bytes.
Re:Xz by huhtenberg on Thursday June 24, 2010 @07:05PM (#1459440)
Ugh. Can you try with a lighter compression level? There is a command line switch for that, and it should also show the memory requirements for each compression level.

I'm insisting because PAQ is a really incredible collection of compressors. Far more intelligence in it than in the LZ* bunch.