News for hackers. Stuff that matters.
Re:MapReduce for the Masses: Zero to Hadoop in 5 minutes wit... by seiji on Friday December 16, 2011 @04:26PM (#3362408)
Read http://infolab.stanford.edu/~ullman/mmds.html a few times and you'll know more about manipulating large data sets than most people.

If you're just looking for a quick example: imagine you want to parse 6,000 webserver logs (each being 100+GB) to determine the most popular pages, referrers and user agents. The map phase extracts all fields you want and the reduce phase counts all the extracted unique values.

"mapreduce" is about processing at scale. You can always do the same thing on a smaller set of data locally (and usually with just sed, awk, sort, and uniq).

There are no comments on this comment yet.