News for hackers. Stuff that matters.
Re:Announcing Spark 1.6 by IvanVergiliev on Monday January 04, 2016 @05:55PM (#10839398)
Reduce can perform reductions on locally on each machine before shuffling the data. This decreases the memory as well as the network overhead. If you need all the elements for a given key - e.g. to display them to a user or save them to a DB, perhaps you should use groupBy. If you're going to perform some form of a reduce after that though, it's likely sub-optimal.

There are no comments on this comment yet.