Re:Full-history English Wikipedia dump produced: 5.6TB uncom...
by _delirium on Wednesday April 14, 2010 @01:24PM
(#1265452)
A lot of researchers just stream the decompressed stream directly from 7z into their analysis scripts. If you were to actually decompress to a giant XML file first, you'd both: 1. need a 6-TB drive; and 2. start getting disk I/O as a big bottleneck. In a lot of cases, the analysis scripts, rather than decompression, are the bottleneck anyway: 7z can feed you data faster than your XML parser and scripts can consume it.
That, and hopefully you could decompress just what you needed. I.E. a tarball of compressed articles.