News for hackers. Stuff that matters.
Re:LinkedIn's Data Infrastructure by KrisJordan on Thursday August 05, 2010 @10:20AM (#1578109)
Surprised to read LinkedIn's "People You May Know" MapReduce pipeline takes 82 jobs.

From what I've read Google's index runs in around 20 MapReduce jobs.

Re:LinkedIn's Data Infrastructure by rjurney on Thursday August 05, 2010 @03:26PM (#1579046)
Pagerank is a like a markov chain - you iterate the same dataset over and over until you're happy with the result. If 20 is good enough, its good enough. Good explanation here: http://www.iterativemapreduce.org/samples.html#Pagerank

Whereas if you include signals from multiple sources, the joins are each one MR job, never mind the calculations.