Re:A New Analytics Toolbox with Apache Spark – Going Beyond ...
by mrbonner on Tuesday August 26, 2014 @02:32PM
(#8228714)
"All in-memory".. yeah and I have 1TB data file to process. How is that going to help?
[ Parent
| on HN ]
Re:A New Analytics Toolbox with Apache Spark – Going Beyond ...
by tupshin on Tuesday August 26, 2014 @02:39PM
(#8228750)
1) Buy a box with 1TB of RAM. Very doable these days, albeit a bit pricey still
2) Scale out. Spark can easily handle hundreds of nodes in a single cluster. Aggregate RAM across all of them can be used.
3) Cache intermediate data sets and/or hot data sets, as opposed to the entire data set.
Re:A New Analytics Toolbox with Apache Spark – Going Beyond ...
by ironchef on Tuesday August 26, 2014 @03:51PM
(#8229130)
For #2 and #3, see here: http://spark.apache.org/docs/latest/programming-guide.html#r...
We typically do #2 at our company and it's been fine so far. The bigger issue isn't a single 1 TB data set. It's multiple large data sets as one then must handle the data shuffles during joins, etc. The ability to keep the RDD in memory through the operations still tends to beat normal long winded hadoop operations anyways...
Re:A New Analytics Toolbox with Apache Spark – Going Beyond ...
by rs_atl on Tuesday August 26, 2014 @04:32PM
(#8229358)
Long winded in more ways than one. I would still use Spark even if it were slower than Hadoop, just to get a sane API.
Re:A New Analytics Toolbox with Apache Spark – Going Beyond ...
by sitkack on Tuesday August 26, 2014 @03:25PM
(#8228976)
256GB machines are the sweet spot right now.
Re:A New Analytics Toolbox with Apache Spark – Going Beyond ...
by rs_atl on Tuesday August 26, 2014 @03:33PM
(#8229031)
It can also spill to disk if needed, which is what Hadoop always does.
Re:A New Analytics Toolbox with Apache Spark – Going Beyond ...
by mike_ivanov on Tuesday August 26, 2014 @05:14PM
(#8229600)
Uh, not actually. In theory - yes, but not in reality. Try to join two large RDDs and spill the result, you'll see what I mean.
Re:A New Analytics Toolbox with Apache Spark – Going Beyond ...
by ConnorBoyd on Tuesday August 26, 2014 @04:03PM
(#8229194)
10 nodes with 100GB each. 100 nodes with 10GB each. 1000 nodes with 1GB each. Spark is built for scaling out.