News for hackers. Stuff that matters.
Re:On the Computational Complexity of MapReduce by chillacy on Sunday October 05, 2014 @02:41PM (#8413025)
MR is a pretty inflexible paradigm, and doesn't really suit graph problems well at all. Most people I know prefer to use Spark (or in the case of graph problems, giraph). Of course this is for huge graphs that won't fit on a machine (petabytes). If it fits on a machine it's far easier to not muck around with distributed computing.
Re:On the Computational Complexity of MapReduce by TallGuyShort on Sunday October 05, 2014 @11:11PM (#8414161)
You may be interested in GraphX, which describes itself as a hybrid between the "data-parallel" approach of MapReduce and core Spark, and the "graph-parallel" approach of Giraph, Graphlab, etc. See http://spark.apache.org/graphx/. It allows you to do graph processing in Spark, and I quite like its approach.
Re:On the Computational Complexity of MapReduce by ihodes on Sunday October 05, 2014 @08:17PM (#8413870)
Do you know of a group using Spark on petabytes (or even a terabyte(s)) of data?
Re:On the Computational Complexity of MapReduce by TallGuyShort on Sunday October 05, 2014 @11:17PM (#8414175)
The scalability of Spark is certainly not as established as that of Hadoop. It's entirely anecdotal as I'm not in a position to share verifiable details, but I know of several installations that run Spark on several-hundred-node clusters. They're certainly well into terabytes, if not in the neighborhood of petabytes.
Re:On the Computational Complexity of MapReduce by rxin on Monday October 06, 2014 @01:15AM (#8414377)
There are several use cases that I know of. One public one is Taobao http://databricks.com/blog/2014/08/14/mining-graph-data-with...