News for hackers. Stuff that matters.
Re:Apache Spark 1.0.0 by iskander on Friday May 30, 2014 @04:43PM (#7823967)
Well, the RDD is initially partitioned using a RangePartitioner over a dense key space of Longs. Each element is then expanded ~100x (each object is significantly smaller than the original value). So the total memory footprint and skew of the expanded RDD shouldn't, theoretically, be a problem.

There are no comments on this comment yet.