News for hackers. Stuff that matters.
Re:Apache Spark 1.0.0 by iskander on Friday May 30, 2014 @04:11PM (#7823741)
So the central piece of data is something like a 10 million element RDD of (UserId, (MovieId, Rating))? If so, it sounds like that data would fit into a single in-memory sparse array, how does Spark's performance compare with a local implementation?

By comparison, I'm trying (and failing) to work with RDDs of 100+ billion elements.

Re:Apache Spark 1.0.0 by platz on Friday May 30, 2014 @04:25PM (#7823842)
What is the difference between Spark and Storm? They both seem like "realtime compute engines"

*edit - from what I can see Spark is a replacement for hadoop (offline jobs), where Storm deals with online stream processing

Re:Apache Spark 1.0.0 by ironchef on Friday May 30, 2014 @05:31PM (#7824209)
Storm is generally more of a dataflow "per event" real/near time computation system (with each event flowing through N spouts and bolts) whereas Spark is more of an in-memory data processing system (with Spark streaming being the "equivalent" to the storm system).