Ask HN: To everybody who uses MapReduce: what problems do you solve?

Re:Ask HN: To everybody who uses MapReduce: what problems do... by virtuabhi on Sunday November 10, 2013 @02:27PM (#6708019)

Yes, people in sciences write distributed code (e.g., using MPI) and employ job queueing systems. I think they work well for embarrassingly parallel tasks. But what if you need to do a group-by operation? MapReduce makes these operations easy. Also the Hadoop system takes care of failing nodes, data partitioning, communication, etc.

I do hope physics people consider MR systems in addition to the existing ones they use.

[ Parent | on HN ]

Re:Ask HN: To everybody who uses MapReduce: what problems do... by apw on Sunday November 10, 2013 @05:03PM (#6708648)

MPI has a reduce operator. On some supercomputers, the operator is implemented in an ASIC.

It is quite easy to implement map/reduce style programs in MPI. It is quite difficult or impossible to implement the parallel algorithms that win the Gordon Bell prize, almost all of which are implemented in MPI, using Hadoop.

[ Parent | on HN ]

Re:Ask HN: To everybody who uses MapReduce: what problems do... by seanmcdirmid on Sunday November 10, 2013 @05:14PM (#6708688)

MPI can't schedule to optimize IO/data shuffling like MapReduce, this is the price for being more general. The biggest competition to MPI in the HPC space is actually CUDA and GPUs. In either case, the data pipeline is exposed to the compiler or run time for optimization, where as MPI everything is just messaging.

[ Parent | on HN ]