Re:Map Reduce: A really simple introduction
by iskander on Thursday September 30, 2010 @01:40PM
(#1744120)
When I was using MapReduce for "real work" it was mostly to count n-grams. I don't think I was the only one...
[ Parent
| on HN ]
Re:Map Reduce: A really simple introduction
by jbooth on Thursday September 30, 2010 @03:12PM
(#1744445)
Big counting problems naturally make a lot of sense for MR, join the like keys in the reduce and add them up, it's beautifully parallel the whole way through. So they're the canonical example.
More complex logic happens of course - but problems that "fit" for MR tend to follow a similar pattern to counting.
Re:Map Reduce: A really simple introduction
by duhprey on Thursday September 30, 2010 @08:16PM
(#1745418)
It's too bad because it could take one more step and do something slightly cooler.
I just do counting in MapReduce, but it was to use Expectation-Maximization on mixtures of Gaussians to cluster a data set of 16 million documents. Although computing the Guassian's isn't really trivial, all you do at the end is sum up and normalize.
It's very cool and isn't much more than counting (replace the map step). In fact the paper I took it from (and the whole Mahout on Hadoup thing) is a bunch of machine learning algorithms that are just big summations at the reduce step.