Re:Full-history English Wikipedia dump produced: 5.6TB uncom...
by _delirium on Wednesday April 14, 2010 @02:12PM
(#1265632)
> I wonder how big a content snapshot is, ie no article histories and no meta-material like talk pages or WP:xxx pages, just the user-facing content
I don't know how big it is uncompressed, but they do have a dump of just that part:
2010-03-16 08:44:40 done Articles, templates, image descriptions, and primary meta-pages.
2010-03-16 08:44:40: enwiki 9654328 pages (255.402/sec), 9654328 revs (255.402/sec), 82.9% prefetched, ETA 2010-03-17 03:08:26 [max 26568677]
This contains current versions of article content, and is the archive most mirror sites will probably want.
pages-articles.xml.bz2 5.7 GB
Re:Full-history English Wikipedia dump produced: 5.6TB uncom...
by anigbrowl on Wednesday April 14, 2010 @02:23PM
(#1265664)
Well spotted. This has great possibilities for education in the 3rd world.
I was also sort of hoping to see from the stats what proportion of content was public-facing vs devoted to arguments between wikipedians...if you look at the stats for 'most edited articles' (accessible from the top link) it's interesting that of the top 50 most edited articles, only one, 'George W. Bush' is user-facing - and I suspect that only made it in because of persistent vandalism.
Still, with history and all included, there is some fabulous data-mining potential here, with which there's the potential to do some really innovative work. I'd hazard a guess that the size of Wikipedia already exceeds that of existing language corpuses like the US code...
/retreats into corner muttering about semantic engines and link free concepts of total hypertext as necessary AI boot conditions