News for hackers. Stuff that matters.
Re:Full-history English Wikipedia dump produced: 5.6TB uncom... by blhack on Wednesday April 14, 2010 @12:06PM (#1265267)
I imagine that the compression is only for saving time during transfer.

That, and hopefully you could decompress just what you needed. I.E. a tarball of compressed articles.

Re:Full-history English Wikipedia dump produced: 5.6TB uncom... by _delirium on Wednesday April 14, 2010 @01:24PM (#1265452)
A lot of researchers just stream the decompressed stream directly from 7z into their analysis scripts. If you were to actually decompress to a giant XML file first, you'd both: 1. need a 6-TB drive; and 2. start getting disk I/O as a big bottleneck. In a lot of cases, the analysis scripts, rather than decompression, are the bottleneck anyway: 7z can feed you data faster than your XML parser and scripts can consume it.