Re:Announcing Spark 1.6
by IvanVergiliev on Monday January 04, 2016 @05:55PM
(#10839398)
Reduce can perform reductions on locally on each machine before shuffling the data. This decreases the memory as well as the network overhead.
If you need all the elements for a given key - e.g. to display them to a user or save them to a DB, perhaps you should use groupBy. If you're going to perform some form of a reduce after that though, it's likely sub-optimal.
[ Parent
| on HN ]
There are no comments on this comment yet.