Re:Ask HN: Are there any purely keyword-based search engines...
by WalterGR on Sunday March 25, 2018 @12:44PM
(#16673003)
Also, did he mean the entire revision history of Wikipedia, or just the most recent versions of pages?
[ Parent
| on HN ]
Re:Ask HN: Are there any purely keyword-based search engines...
by giancarlostoro on Sunday March 25, 2018 @12:52PM
(#16673051)
That would definitely explain 2TB haha but yeah, if you pull the recent English version without images it's like ... I wanna say 30GB off the top of my head.
As for anyone curious:
Re:Ask HN: Are there any purely keyword-based search engines...
by 3131s on Sunday March 25, 2018 @01:47PM
(#16673356)
The most recent was 13.9GB compressed and ~65GB uncompressed. The version without images, talk pages, and revision history is 'enwiki-20180320-pages-articles.xml.bz2' here:
https://dumps.wikimedia.org/enwiki/20180320/
The article text contains Wikipedia markup which is a bit difficult to remove but not impossible, there are some existing projects for doing that. DBPedia would have the raw text, but it's not nearly as current.
Re:Ask HN: Are there any purely keyword-based search engines...
by lacksconfidence on Sunday March 25, 2018 @04:14PM
(#16674305)
You can get the raw text with wikitext and whatnot removed from the search dumps made every week (also has various other metadata)
http://dumps.wikimedia.your.org/other/cirrussearch/
These are structured with a JSON string for each doc roughly like https://en.m.wikipedia.org/wiki/Foobar?action=cirrusdump