News for hackers. Stuff that matters.

A comment on How Akka Streams can be used to process the Wikidata dump in parallel.

Re:How Akka Streams can be used to process the Wikidata dump... by mtrn on Tuesday June 23, 2015 @09:30AM (#9764467)
On a related note: When I indexed the whole English wikipedia last year, I was surprised, that it was possible to have a JSON version of it indexed[1] and searchable within half an hour on my laptop.

[1] Using parallel bulk indexer for ES: https://github.com/miku/esbulk

[ Parent | on HN ]
Re:How Akka Streams can be used to process the Wikidata dump... by mhuffman on Tuesday June 23, 2015 @10:44AM (#9764918)
How about ~17 minutes (including wikipedia data download and extraction time)! Using json-wikipedia and lbzip2

[1] https://github.com/diegoceccarelli/json-wikipedia

[ Parent | on HN ]
Re:How Akka Streams can be used to process the Wikidata dump... by mtrn on Tuesday June 23, 2015 @11:00AM (#9765036)
Thanks, JSON exports make wikipedia data much more approachable.
[ Parent | on HN ]
Re:How Akka Streams can be used to process the Wikidata dump... by andrewvc on Tuesday June 23, 2015 @04:32PM (#9767579)
I wrote a similar thing with an emphasis on multiple analyzed fields (making it slower to index) but much more flexible to query.

https://github.com/andrewvc/wikiparse/tree/java

That being said, when it comes to indexing wikipedia, the indexing can be done well across multiple threads internally by elasticsearch. Multithreading the reading/parsing isn't a huge win. Doing decompression in a separate thread is however.

[ Parent | on HN ]
Re:How Akka Streams can be used to process the Wikidata dump... by mtrn on Tuesday June 23, 2015 @05:16PM (#9767864)
Yes, ES uses multiple threads nicely. But as you move to 32 or 64 cores - in my experience - a single threaded client won't keep ES/Lucene busy enough.

With SOLR, it's similar:

> Sometimes you need to index a bunch of documents really, really fast. [...] The solution is two-fold: batching and multi-threading

From: http://lucidworks.com/blog/high-throughput-indexing-in-solr/

[ Parent | on HN ]