Re:Wikipedia data dumps and stats
by fauigerzigerk on Sunday March 24, 2013 @03:29AM
(#5431402)
Sadly, they don't publish up-to-date HTML dumps and there is no reliable way of reproducing them short of installing the entire wikipedia system locally, including the database. I know there are quite a few projects that claim to do it but they're all abandoned, incomplete or unsuitable in various other ways (as far as I know).
[ Parent
| on HN ]
Not sure if this helps, but there's the Kiwix offline Wikipedia reader and its associated Zim files:
http://download.kiwix.org/zim/0.9/
http://www.kiwix.org/wiki/Tools/en#Generating_ZIM_Files_From...
"The entire Wikipedia system, including the database" is slightly overblown: it's just MediaWiki and MySQL.
MediaWiki (+ a finely tuned PHP system, because mediawiki is unusably slow without it), MySQL, and plenty of time / resources for doing the large import of enwiki (since that is what most people are interested in).
Re:Wikipedia data dumps and stats
by fauigerzigerk on Sunday March 24, 2013 @05:45AM
(#5431576)
Exactly, and at that point you still don't have the static HTML files. You have to crawl the entire local site, which takes ages. Then you have to repeat all of this according to your desired update frequency.