News for hackers. Stuff that matters.
Re:Xz by spicyj on Thursday June 24, 2010 @02:12PM (#1458651)
Can someone explain to me why this Wikipedia article is at the top of the front page?
Re:Xz by wmf on Thursday June 24, 2010 @02:16PM (#1458663)
Most hackers don't know about XZ yet, but they should. I still prefer not to see Wikipedia articles on HN since they lack context. A "7 reasons you should ditch bzip2 and use XZ" blog post would probably be better.
Re:Xz by someone_here on Thursday June 24, 2010 @02:17PM (#1458667)
Oh god please not more "X reasons for Y"
Re:Xz by WilliamLP on Thursday June 24, 2010 @02:27PM (#1458702)
Give me Z reasons why you don't like "X reasons for Y" posts.
Re:Xz by SkyMarshal on Thursday June 24, 2010 @02:36PM (#1458734)
9 Reasons Why You Should Hate "X Reasons for Y" posts:

1. They're overdone

2. They're uncreative

3. They're unoriginal

4. They're a crutch

5. They're assembly-line writing

6. They're used by tabloids to appeal to supermarket zombies

7. They've helped turn respectable mags into tabloids (or are a symptom of it, not sure which came first)

8. They're psychologically manipulative (I don't know how, I just know it)

9. They work

:)

Re:Xz by joshfinnie on Thursday June 24, 2010 @03:16PM (#1458847)
You forgot to put these across 11 individual comments so your 8pts would have been multiplied to 88!

I do not mind lists, but I cannot stand lists that are seperated to pages for inflated page counts!

Re:Xz by koeselitz on Thursday June 24, 2010 @04:27PM (#1459016)
He was going to put it across 11 comments, but as the Wikipedia article clearly points out, xz compression conventionally bundles a single file.
Re:Xz by Perceval on Thursday June 24, 2010 @04:01PM (#1458962)
The HN Guidelines explicitly frown upon "X reasons..." titles because they're almost always linkbait.

If the original title begins with a number or number + gratuitous adjective, we'd appreciate it if you'd crop it. E.g. translate "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to "Ys." Exception: when the number is meaningful, e.g. "The 5 Platonic Solids."

http://ycombinator.com/newsguidelines.html

Re:Xz by ramchip on Thursday June 24, 2010 @06:41PM (#1459379)
How about a "Y is the new X" or "X is dead, long live Y"?
Re:Xz by tvon on Thursday June 24, 2010 @04:03PM (#1458971)
I'd prefer it to a two paragraph Wikipedia article.
Re:Xz by snippyhollow on Thursday June 24, 2010 @02:35PM (#1458727)
"X reasons for Z": there, I fixed it for you.
Re:Xz by ElliotH on Thursday June 24, 2010 @02:23PM (#1458688)
Agree on the wikipedia articles with no context point. Especially since the article in question is short with not very much real information on why it matters.
Re:Xz by matthavener on Thursday June 24, 2010 @02:17PM (#1458664)
LZMA2 has a better compression ratio than bzip2, according to this article http://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Mark... . Seems like it would be a good idea to switch away from bzip2/gzip to zx
Re:Xz by wheels on Thursday June 24, 2010 @02:23PM (#1458687)
The main reason to continue to use gzip is speed: it's an order of magnitude faster than bzip2 for compression / decompression, for a relatively small compression ratio hit. It's fast enough to us inline in web servers, SSH sessions, etc, and if you're just moving data around a single time, often the amount of time for compress / transfer / decompress is much faster than with bzip2.
Re:Xz by gxti on Thursday June 24, 2010 @04:33PM (#1459034)
I've found that on many types of data, xz -2 is twice as fast and compresses better than gzip -9. Of course gzip -1 kicks the pants off xz but there's absolutely no reason to use gzip at its higher levels, nor to use bzip2 at all. And if you want fast, LZO does it better -- the lzop tool provides a typical unix compress interface.

Using 50MiB from a tarball I had lying around, best of 3:

     prog    time(s)   size(%)
    gzip  -1  1.703    38.7
    gzip  -9 14.670    35.1    (worse than xz -1)
    bzip2 -1  7.714    34.0
    bzip2 -9  8.035    31.4    (worse than xz -1)
    xz    -1  6.278    31.3
    xz    -9 42.445    22.7    (best compression)
    lzop  -1  0.670    46.1    (fastest)
    lzop  -9 18.144    38.0
Re:Xz by gxti on Friday June 25, 2010 @09:58AM (#1460733)
If anyone is still reading this thread, I made a chart overnight using a better corpus:

http://partiallystapled.com/~gxti/misc/2010/06/25-shootout.p...

http://partiallystapled.com/~gxti/misc/2010/06/25-linux-2.6....

Re:Xz by huhtenberg on Thursday June 24, 2010 @05:34PM (#1459205)
Out of curiousity, can you run this tarball through paq (say, paq8hp12)?

http://en.wikipedia.org/wiki/PAQ

Re:Xz by gxti on Thursday June 24, 2010 @06:39PM (#1459374)
It segfaulted at the end, after taking 7m28s. I don't know how big the result was because it got truncated to 35 bytes.
Re:Xz by huhtenberg on Thursday June 24, 2010 @07:05PM (#1459440)
Ugh. Can you try with a lighter compression level? There is a command line switch for that, and it should also show the memory requirements for each compression level.

I'm insisting because PAQ is a really incredible collection of compressors. Far more intelligence in it than in the LZ* bunch.

Re:Xz by alecco on Thursday June 24, 2010 @03:22PM (#1458862)
Yes, the new compression algorithms (BWT and LZ77-based) are slower and require a lot more memory than DEFLATE (*zip) in most cases.

This guy did a not so rigorous analysis but it is mostly OK: http://changelog.complete.org/archives/931-how-to-think-abou...

A little bird told me there's a better algorithm in speed/compression rate coming soon ;)

Re:Xz by kunley on Friday June 25, 2010 @10:34AM (#1460828)
Another feature of gzip which may be your life-saver is that it keeps a record of the original size so you can know it without decompression.

xz format seems to have this feature, but: $ xz --list xz: --list is not implemented yet.

Re:Xz by JoachimSchipper on Thursday June 24, 2010 @02:54PM (#1458792)
More importantly, xz decompresses quickly. Essentially, you get the compression ratio of bzip2 at the decompression speed of gzip (and, unfortunately, the compression speed of bzip2).

EDIT: also, like gzip but unlike bzip2, you can stream data through xz (at an insignificant penalty in compression ratio).

Re:Xz by gaius on Thursday June 24, 2010 @02:41PM (#1458757)
LZMA is much, much slower. It is simply a matter of the rate at which you generate data and the compression ratio and the rate at which you add more storage/move the files off to tape. Can you "keep up" essentially.

Right now, CPUs are fast enough that LZMA is a realistic prospect. It boils down to what the extra CPUs will cost vs the cost of the storage saved (incl. the "cost" of space in the datacentre, the administrative overhead of more hardware, etc).

Re:Xz by stcredzero on Thursday June 24, 2010 @04:03PM (#1458968)
I wonder if it would be possible to have page templates delivered separately from their dynamic content? The templating of the dynamic data would happen on the browser-side machine. This way, we could use really expensive compression algorithms with high CPU cost for compression, but then cache the compression result.

This could be achieved by using Javascript and fetching data as JSON, but it seems to be something that would be very useful and beneficial if it were standardized.

Re:Xz by jhy on Thursday June 24, 2010 @06:09PM (#1459300)
There is the SDCH spec: Shared Dictionary Compression over HTTP. Google Chrome uses it, and the Google Toolbar adds support to IE.

The server supplies a pre-created dictionary (template) which gets cached, and further requests get a diff to that dictionary as a response.

I've only found Google search using it on the server side, and the browser penetration is fairly low, but it seems promising. It makes requests very fast.

I've also noticed that Google search serves image thumbnails as data:// URLs directly in the original response. They are going a long way to ensuring each page loads entirely in one request.

Re:Xz by pornel on Thursday June 24, 2010 @05:40PM (#1459224)
There is client-side XSLT already. The problem is that transformed template is usually same size or smaller than your source data.

Really expensive algorithms aren't that much better to offset cost of sending template logic to the client.

You could probably save some processing time by integrating compression directly into server-side templating - take advantage of the fact that some parts never change and keep them pre-compressed or at least cache some statistics about them to aid compression.

Re:Xz by golwengaud on Thursday June 24, 2010 @02:28PM (#1458707)
In retrospect, I probably should have submitted the XZ Utils project page: http://tukaani.org/xz/ .

However, that page does not give the same picture of widespread adoption that the Wikipedia article does. That use was what interested me: I'm surprised that it only came to my attention when I was trying to recall the name of the 7z command line utility p7zip.

Re:Xz by goodside on Thursday June 24, 2010 @05:37PM (#1459219)
I upvoted it because, on top of being interesting, it fights the passively accepted wisdom that "newsworthy" is synonymous with "worthy".
Re:Xz by lanstein on Thursday June 24, 2010 @02:14PM (#1458656)
I'm guessing because people voted it up. Additionally, it's not a peak time right now.
Re:Xz by mechanician on Thursday June 24, 2010 @02:14PM (#1458657)
When is peak time?
Re:Xz by kiba on Thursday June 24, 2010 @02:21PM (#1458681)
It's apparently something new that has been floating around. Some hackers are probably curious about the "new thing" that popped up lately.
Re:Xz by scdlbx on Thursday June 24, 2010 @02:22PM (#1458684)
Software that was released 9 months ago isn't particularly new.
Re:Xz by joshfinnie on Thursday June 24, 2010 @03:18PM (#1458853)
First, I think that is why it's in quotes.

Second, I am reading Of Mice or Men for the first time this summer. It is new to me!