News for hackers. Stuff that matters.
Re:How Akka Streams can be used to process the Wikidata dump... by frik on Tuesday June 23, 2015 @11:39AM (#9765369)
> Process the whole Wikidata in 7 minutes with your laptop

Wikidata is several magnitudes smaller than Freebase (closed by Google in May) and it won't fit in your RAM (laptop).

Re:How Akka Streams can be used to process the Wikidata dump... by xomateix on Tuesday June 23, 2015 @11:53AM (#9765462)
As you comment, Freebase is bigger than Wikidata. It is 22GB compressed (250GB uncompressed) while Wikidata is 5GB compressed (49GB uncompressed) [1].

Said that, I believe the process described in the blog post is not loading the whole Wikidata dump into memory and it would work the same to process Freebase or even larger data dumps with your laptop.

From the post: How Akka Streams can be used to process the Wikidata dump in parallel and using constant memory with just your laptop.

[1] https://developers.google.com/freebase/data http://dumps.wikimedia.org/other/wikidata/

Re:How Akka Streams can be used to process the Wikidata dump... by thibaut_barrere on Tuesday June 23, 2015 @11:46AM (#9765419)
What are your favorite large, publicly available datasets?
Re:How Akka Streams can be used to process the Wikidata dump... by Smerity on Tuesday June 23, 2015 @05:19PM (#9767880)
Biased reply (I'm a data scientist there): Common Crawl[1]. We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone completely free.

[1]: http://commoncrawl.org/

Re:How Akka Streams can be used to process the Wikidata dump... by rcpt on Tuesday June 23, 2015 @11:19PM (#9769344)
Re:How Akka Streams can be used to process the Wikidata dump... by nextos on Tuesday June 23, 2015 @01:12PM (#9766033)
The Cancer Genome Atlas, Ensembl, 1000Genomes.