Re:Bagel: Large-scale graph processing on Spark
by HairyFotr on Wednesday June 22, 2011 @05:00AM
(#2682328)
Is Spark really as fast in real use, as the graph on the site claims? I don't think I've ever seen anything beat Hadoop by that margin before.
Re:Bagel: Large-scale graph processing on Spark
by danjo on Thursday June 23, 2011 @02:24AM
(#2686894)
We currently use Spark for generating reports that scan the same data set multiple times. We got a 20-30x performance gain over hadoop+hive. Spark's ability to keep data sets in memory works very well for our use case.
- Support map-reduce style computation model (i.e. map, reduce, groupByKey style computations). It is important to differentiate map-reduce infrastructure (e.g. Google MapReduce, Hadoop MapReduce) from the computation model (e.g. map/reduce function).
- For many use cases, minimal change required going from serial code to distributed, parallelized code.
- Very nice REPL interface that's very convenient in exploring data.
- Much better support for iterative (machine learning) computations through in-memory cache of data sets and minimal per iteration scheduling overhead.
- Good integration with existing infrastructures, e.g. Hadoop HDFS
To find out more about the framework: http://www.spark-project.org/