Re:Apache Spark 1.0.0
by iskander on Friday May 30, 2014 @04:43PM
(#7823967)
Well, the RDD is initially partitioned using a RangePartitioner over a dense key space of Longs. Each element is then expanded ~100x (each object is significantly smaller than the original value). So the total memory footprint and skew of the expanded RDD shouldn't, theoretically, be a problem.
[ Parent
| on HN ]
There are no comments on this comment yet.