News for hackers. Stuff that matters.
Re:Full-history English Wikipedia dump produced: 5.6TB uncom... by jonknee on Wednesday April 14, 2010 @11:53AM (#1265222)
The article says Tb not TB, but in reality it appears to be TB. That's quite a difference. Still seems heavy for text, but I assume the full text of every revision is in it, not just diffs.
Re:Full-history English Wikipedia dump produced: 5.6TB uncom... by _delirium on Wednesday April 14, 2010 @01:37PM (#1265494)
Yeah, every revision is standalone, which is why it compresses so well (obviously there are a lot of edits that make relatively small changes). One reason is to make it easier for researchers to grab specific revisions, e.g. run the dump through a filter returning only revisions as of June 1, 2006--- without having to apply a ton of diffs to reconstruct those revisions.

The dump schema is something like:

  <mediawiki blah blah ...>
    <siteinfo>
      some metadata
    </siteinfo>
    <page>
      <title>Article Title</title>
      <id>15580374</id>
      <revision>
        <id>139992</id>
        <timestamp>2002-01-26T15:28:12Z</timestamp>
        <contributor>
          <username>_delirium</username>
          <id>82</id>
        </contributor>
        <comment>vandalized this page</comment>
        <text xml:space="preserve">Complete text of this revision of the article goes here.
        </text>
      </revision>
      <revision>
        ...next revision of this page...
      </revision>
    </page>
    <page>
      ...revisions of the next page...
    </page>
  </mediawiki>