News for hackers. Stuff that matters.
Re:Full-history English Wikipedia dump produced: 5.6TB uncom... by _delirium on Wednesday April 14, 2010 @08:50PM (#1266613)
In the dump, I think for robustness and ease of extracting subsets.

Robustness: Having to essentially play back a log to recover any particular revision increases a chance of something eventually getting corrupted, and so it's somewhat safer to avoid it in something intended to be archival.

Ease of extracting subsets: For researchers, having the revisions be independent allows you to filter the XML dump through a SAX parser (or similar) to grab only revisions meeting particular criteria. If deltas were stored, you'd have to reconstruct those revisions from the deltas, which would make it really expensive to do things like, "I want to look at every article as it appeared at noon on April 1, 2007".

In the live DB, I think just because it's cheaper to get a ton of storage, esp. for rarely-retrieved old revisions, than to add the overhead of computing deltas and applying them to reconstruct revisions. In particular, you'd have to compute a diff for every edit in that situation, whereas currently MediaWiki only computes diffs when a user requests to view one from the "history" tab, which is a tiny proportion of all edits.

Re:Full-history English Wikipedia dump produced: 5.6TB uncom... by JoachimSchipper on Thursday April 15, 2010 @05:32AM (#1267388)
I understand it's simpler to store everything, and simplicity _is_ a virtue; but one could store the current revision plus deltas (and perhaps a few intermediate revisions for oft-edited articles), and obtain performance at least as good as in the current case. It would also save lots of space.