Re:How Akka Streams can be used to process the Wikidata dump...
by xomateix on Tuesday June 23, 2015 @11:53AM
(#9765462)
As you comment, Freebase is bigger than Wikidata. It is 22GB compressed (250GB uncompressed) while Wikidata is 5GB compressed (49GB uncompressed) [1].
Said that, I believe the process described in the blog post is not loading the whole Wikidata dump into memory and it would work the same to process Freebase or even larger data dumps with your laptop.
From the post: How Akka Streams can be used to process the Wikidata dump in parallel and using constant memory with just your laptop.
[1] https://developers.google.com/freebase/data http://dumps.wikimedia.org/other/wikidata/
Re:How Akka Streams can be used to process the Wikidata dump...
by thibaut_barrere on Tuesday June 23, 2015 @11:46AM
(#9765419)
What are your favorite large, publicly available datasets?
Re:How Akka Streams can be used to process the Wikidata dump...
by Smerity on Tuesday June 23, 2015 @05:19PM
(#9767880)
Biased reply (I'm a data scientist there): Common Crawl[1]. We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone completely free.
Re:How Akka Streams can be used to process the Wikidata dump...
by rcpt on Tuesday June 23, 2015 @11:19PM
(#9769344)
This thread is pretty good
http://www.quora.com/Where-can-I-find-large-datasets-open-to...
Wikidata is several magnitudes smaller than Freebase (closed by Google in May) and it won't fit in your RAM (laptop).