How Akka Streams can be used to process the Wikidata dump in parallel

Re:How Akka Streams can be used to process the Wikidata dump... by andrewvc on Tuesday June 23, 2015 @04:32PM (#9767579)

I wrote a similar thing with an emphasis on multiple analyzed fields (making it slower to index) but much more flexible to query.

https://github.com/andrewvc/wikiparse/tree/java

That being said, when it comes to indexing wikipedia, the indexing can be done well across multiple threads internally by elasticsearch. Multithreading the reading/parsing isn't a huge win. Doing decompression in a separate thread is however.

[ Parent | on HN ]

Re:How Akka Streams can be used to process the Wikidata dump... by mtrn on Tuesday June 23, 2015 @05:16PM (#9767864)

Yes, ES uses multiple threads nicely. But as you move to 32 or 64 cores - in my experience - a single threaded client won't keep ES/Lucene busy enough.

With SOLR, it's similar:

> Sometimes you need to index a bunch of documents really, really fast. [...] The solution is two-fold: batching and multi-threading

From: http://lucidworks.com/blog/high-throughput-indexing-in-solr/

[ Parent | on HN ]