News for hackers. Stuff that matters.
Re:Spark: Open Source Superstar Rewrites Future of Big Data by justinsb on Wednesday June 19, 2013 @02:27PM (#5907295)
Can someone explain why the in-memory caching is such a big win? Does Hadoop MapReduce not do caching as well? I'd expect at least filesystem caching when the computation is running on the same machine as the data block...
Re:Spark: Open Source Superstar Rewrites Future of Big Data by rxin on Wednesday June 19, 2013 @02:48PM (#5907414)
Spark goes much more beyond just in-memory caching. It features a more advanced scheduler and a higher level programming abstraction.

The programming abstraction treats all data as collections (RDD in Spark terminology), and allow programmers to apply bulk transformation on these collections. Some examples of operations you can apply include the traditional map and reduce, the relational filter, join, outerJoin, and more advanced ones like sample. This abstraction makes it much easier to write distributed programs. As the Wired article mentioned, a distributed program written in Spark often looks identical to a single node program. This substantially reduces the amount of code one needs to write for distributed programs, and the best part is the code really expresses the algorithm (rather than cluttered with JobConf setup).

And the scheduler and the engine itself are aware of the general DAG of operators, so they can schedule and run those operators better. For example, if you have multiple maps, the execution gets pipelined; if you are joining two collections that are partitioned the same way, the execution avoids an expensive shuffle step.

There are many other benefits too. I'd encourage you to give a try. Thanks!

Disclaimer: I am on the Spark team at UC Berkeley.

Re:Spark: Open Source Superstar Rewrites Future of Big Data by HCIdivision17 on Wednesday June 19, 2013 @05:31PM (#5908441)
Does Spark address the same problems as Storm?

It doesn't look like there's been any direct comparison of the two, though it looks like there's overlap. (I've wanted to start a streaming data processing project, and this looks like it would be good to consider for it.)

Re:Spark: Open Source Superstar Rewrites Future of Big Data by krcz on Wednesday June 19, 2013 @06:46PM (#5908940)
There are some nice looking Spark vs Storm graphs in their streaming presentation slides: http://spark-project.org/talks/strata_spark_streaming.pdf . Makes me wonder how biased these might be.
Re:Spark: Open Source Superstar Rewrites Future of Big Data by ikhahmedov on Wednesday June 19, 2013 @06:09PM (#5908726)
Last versions of Spark has ability to do stream processing. http://spark-project.org/docs/latest/streaming-programming-g...
Re:Spark: Open Source Superstar Rewrites Future of Big Data by justinsb on Wednesday June 19, 2013 @03:01PM (#5907503)
Thanks - a nice explanation of Spark's benefits. The programming & execution models sound like big improvements - I'll check it out!
Re:Spark: Open Source Superstar Rewrites Future of Big Data by joshhart on Wednesday June 19, 2013 @10:16PM (#5909775)
Justin: 1. Hadoop is multi-tenant. Another job could burst your cache. Unless things have dramatically changed since I first read the source a year or two ago, spark jobs basically grab control of the cluster so that the cache won't get evicted. 2. Think of Pregel and other iterative-style jobs. You may run several iterations before writing to disk to checkpoint. This can speed up your job immensely since you don't have to write & read data from slow disks. Of course you lose on fault tolerance but the speedup may be worth it.
Re:Spark: Open Source Superstar Rewrites Future of Big Data by mthomas on Wednesday June 19, 2013 @02:34PM (#5907344)
My understanding of hadoop and hdfs is that both input and ourput are to/from disk. Your MR jobs are spun up and pointed to hdfs urls to read input from and they write to hdfs when done. This means that when there is an error, the intermediate computations don't necessarily have to be redone. However there is a trade off.

Additionally, I believe that HDFS keeps 3 copies of the data around on 3 different nodes for redundancy. So there is the overhead of that network traffic.

Re:Spark: Open Source Superstar Rewrites Future of Big Data by justinsb on Wednesday June 19, 2013 @02:58PM (#5907478)
I did find this (now-fixed) bug/enhancement: https://issues.apache.org/jira/browse/HDFS-2246

Sounds like, with that configuration applied, the in-memory performance difference between Hadoop and Spark should not be nearly as large.

Re:Spark: Open Source Superstar Rewrites Future of Big Data by hobbyist on Wednesday June 19, 2013 @07:15PM (#5909111)
Good question. I did read the spark paper, and one reason that I found for spark doing so much better than hadoop was that it avoids the unnecessary serialization, deserialization which hadoop just can not avoid. The RDD's as mentioned by @rxin, are in memory objects and thus do not require frequent serialization/deserialization when multiple operations are being applied to data.
Re:Spark: Open Source Superstar Rewrites Future of Big Data by flatfilefan on Wednesday June 19, 2013 @02:54PM (#5907449)
Unlike linear regression with normally distributed residuals, it is not possible to find a closed-form expression for the coefficient values that maximizes the likelihood function, so an iterative process must be used instead. Wikipedia
Re:Spark: Open Source Superstar Rewrites Future of Big Data by joshhart on Wednesday June 19, 2013 @10:19PM (#5909780)
You can find closed-form expressions for SOME likelihood functions. This is the case for logistic regression.

Anyway, the takeaway that I think you were downvoted for omitting is that if you can bypass the serialization, startup, and shutdown steps associated with the traditional Hadoop iterative process you'll get a huge speedup.