Re:Social Graph Analysis using Elastic MapReduce and PyPy
by aksbhat on Wednesday May 04, 2011 @01:19PM
(#2514506)
I have tried using both Hadoop (55 node cluster at Cornell) and a single AWS High Memory double extra large instance with 32GB memory.
I have found that since the Twitters social graph is small enough to fit in the memory, a single instance with huge amount of RAM is much more efficient, especially when your algorithm iterates over nodes in the network.
You can read about it here:
Hadoop based results:
www.akshaybhat.com/LPMR/
Results using a single High Memory instance AWS instance
www.akshaybhat.com/LPMR/GRAPHLAB
Even the startup hunch has taken a similar approach and use a single machine with large amount of memory rather than a hadoop cluster.
Re:Social Graph Analysis using Elastic MapReduce and PyPy
by mcroydon on Wednesday May 04, 2011 @01:37PM
(#2514587)
Good call. The dataset was large enough that it didn't feel silly to use something like MapReduce but the same thought has been in the back of my head the whole time.
I have found that since the Twitters social graph is small enough to fit in the memory, a single instance with huge amount of RAM is much more efficient, especially when your algorithm iterates over nodes in the network.
You can read about it here:
Hadoop based results: www.akshaybhat.com/LPMR/
Results using a single High Memory instance AWS instance www.akshaybhat.com/LPMR/GRAPHLAB
Even the startup hunch has taken a similar approach and use a single machine with large amount of memory rather than a hadoop cluster.