Re:Full-history English Wikipedia dump produced: 5.6TB uncom...
by jonknee on Wednesday April 14, 2010 @11:53AM
(#1265222)
The article says Tb not TB, but in reality it appears to be TB. That's quite a difference. Still seems heavy for text, but I assume the full text of every revision is in it, not just diffs.
[ Parent
| on HN ]
Re:Full-history English Wikipedia dump produced: 5.6TB uncom...
by _delirium on Wednesday April 14, 2010 @01:37PM
(#1265494)
Yeah, every revision is standalone, which is why it compresses so well (obviously there are a lot of edits that make relatively small changes). One reason is to make it easier for researchers to grab specific revisions, e.g. run the dump through a filter returning only revisions as of June 1, 2006--- without having to apply a ton of diffs to reconstruct those revisions.
The dump schema is something like:
<mediawiki blah blah ...>
<siteinfo>
some metadata
</siteinfo>
<page>
<title>Article Title</title>
<id>15580374</id>
<revision>
<id>139992</id>
<timestamp>2002-01-26T15:28:12Z</timestamp>
<contributor>
<username>_delirium</username>
<id>82</id>
</contributor>
<comment>vandalized this page</comment>
<text xml:space="preserve">Complete text of this revision of the article goes here.
</text>
</revision>
<revision>
...next revision of this page...
</revision>
</page>
<page>
...revisions of the next page...
</page>
</mediawiki>