Re:Full-history English Wikipedia dump produced: 5.6TB uncom...
by ars on Wednesday April 14, 2010 @05:03PM
(#1266147)
I'm pretty sure LZMA (7z) compresses better than bz2.
At least it does when I test it. But it's slower - sometimes much slower (depends on settings).
Re:Full-history English Wikipedia dump produced: 5.6TB uncom...
by leif on Wednesday April 14, 2010 @06:05PM
(#1266280)
That might be true, though I don't know whether that depends on the entropy in the file. It is likely the case that one of them compresses text better but takes a larger performance hit with binary data or somesuch.
Turns out the paper I was recalling and referencing dealt with LZO, not LZMA, so maybe I have less to say about LZMA than I thought. Shows how much you jerks read before upvoting. ;-)
Gzip usually gets a slightly better compression ratio, but at the expense of decompression speed, particularly on less compressible data (LZMA somehow seems to know better when to give up trying to compress). Bzip2 has the best compression ratio of the three, but is far too slow to compress and decompress, so you end up losing more time decompressing than you gained by doing less actual I/O.
EDIT: source, for those curious folk out there: http://portal.acm.org/citation.cfm?id=1534536 (caveat: the experiments were run by taking a large file on disk and compressing it to another large file on disk, so seek thrashing may have been an issue and I'm not quick to take the numbers for all they should be worth)