News for hackers. Stuff that matters.

A comment on Ask HN: Are there any purely keyword-based search engines with no NLP?. Parent comment

Re:Ask HN: Are there any purely keyword-based search engines... by giancarlostoro on Sunday March 25, 2018 @12:04PM (#16672766)
To be fair wikipedia compressed without images is pretty small. A couple tens of gigs no more than 80. You can download it to your phone and there are apps to view it all. Not sure how larger it is decompressed but 2 TB sounds like overkill. But yeah wikipedia would be a good way to test it.

Also did you mean every single wikipedia site or just the English one? I'm thinking just the English one's sufficient.

[ Parent | on HN ]
Re:Ask HN: Are there any purely keyword-based search engines... by tomcooks on Sunday March 25, 2018 @12:19PM (#16672855)
I know what you mean with "sufficient",but English pages are often lacking informations on a lot of subjects related to countries outside the anglosphere.

I still don't understand why wikipedia articles aren't translated according to a common script, at least where there are featured translations (marked a star)

[ Parent | on HN ]
Re:Ask HN: Are there any purely keyword-based search engines... by zeckalpha on Tuesday March 27, 2018 @01:02AM (#16685399)
Feel free to make that happen.
[ Parent | on HN ]
Re:Ask HN: Are there any purely keyword-based search engines... by WalterGR on Sunday March 25, 2018 @12:44PM (#16673003)
Also, did he mean the entire revision history of Wikipedia, or just the most recent versions of pages?
[ Parent | on HN ]
Re:Ask HN: Are there any purely keyword-based search engines... by giancarlostoro on Sunday March 25, 2018 @12:52PM (#16673051)
That would definitely explain 2TB haha but yeah, if you pull the recent English version without images it's like ... I wanna say 30GB off the top of my head.

As for anyone curious:

https://dumps.wikimedia.org/backup-index.html

[ Parent | on HN ]
Re:Ask HN: Are there any purely keyword-based search engines... by 3131s on Sunday March 25, 2018 @01:47PM (#16673356)
The most recent was 13.9GB compressed and ~65GB uncompressed. The version without images, talk pages, and revision history is 'enwiki-20180320-pages-articles.xml.bz2' here:

https://dumps.wikimedia.org/enwiki/20180320/

The article text contains Wikipedia markup which is a bit difficult to remove but not impossible, there are some existing projects for doing that. DBPedia would have the raw text, but it's not nearly as current.

[ Parent | on HN ]
Re:Ask HN: Are there any purely keyword-based search engines... by lacksconfidence on Sunday March 25, 2018 @04:14PM (#16674305)
You can get the raw text with wikitext and whatnot removed from the search dumps made every week (also has various other metadata)

http://dumps.wikimedia.your.org/other/cirrussearch/

These are structured with a JSON string for each doc roughly like https://en.m.wikipedia.org/wiki/Foobar?action=cirrusdump

[ Parent | on HN ]