Ask HN: Are there any purely keyword-based search engines with no NLP?

Re:Ask HN: Are there any purely keyword-based search engines... by giancarlostoro on Sunday March 25, 2018 @12:52PM (#16673051)

That would definitely explain 2TB haha but yeah, if you pull the recent English version without images it's like ... I wanna say 30GB off the top of my head.

As for anyone curious:

https://dumps.wikimedia.org/backup-index.html

[ Parent | on HN ]

Re:Ask HN: Are there any purely keyword-based search engines... by 3131s on Sunday March 25, 2018 @01:47PM (#16673356)

The most recent was 13.9GB compressed and ~65GB uncompressed. The version without images, talk pages, and revision history is 'enwiki-20180320-pages-articles.xml.bz2' here:

https://dumps.wikimedia.org/enwiki/20180320/

The article text contains Wikipedia markup which is a bit difficult to remove but not impossible, there are some existing projects for doing that. DBPedia would have the raw text, but it's not nearly as current.

[ Parent | on HN ]

Re:Ask HN: Are there any purely keyword-based search engines... by lacksconfidence on Sunday March 25, 2018 @04:14PM (#16674305)

You can get the raw text with wikitext and whatnot removed from the search dumps made every week (also has various other metadata)

http://dumps.wikimedia.your.org/other/cirrussearch/

These are structured with a JSON string for each doc roughly like https://en.m.wikipedia.org/wiki/Foobar?action=cirrusdump

[ Parent | on HN ]