Ask HN: To everybody who uses MapReduce: what problems do you solve?

Re:Ask HN: To everybody who uses MapReduce: what problems do... by alecco on Sunday November 10, 2013 @12:01PM (#6707279)

> Does anyone have a use case where data is on a single machine and map reduce is still relevant?

What matters is the running data structure. For example, you can have Petabytes of logs but you need a map/table of some kind to do aggregations/transformations. Or a sparse-matrix based model. There are types of problems that can partition the data structure and work in parallel in the RAM of many servers.

Related: it's extremely common to confuse for CPU bottleneck what's actually a problem of TLB-miss, cache-miss, or bandwidth limits (RAM, disk, network). I'd rather invest time in improving the basic DS/Algorithm than porting to Map/Reduce.

[ Parent | on HN ]

Re:Ask HN: To everybody who uses MapReduce: what problems do... by collyw on Sunday November 10, 2013 @12:12PM (#6707336)

OK, maybe I am not understanding you correctly, but what you describe seems to be, if the data is on one machine, connect to a cluster of machines, and run processing in parallel on that.

That doesn't imply a NoSQL solution to me. Just parallel processing on different parts of the data. If I am wrong can you point me to a clearer example?

[ Parent | on HN ]

Re:Ask HN: To everybody who uses MapReduce: what problems do... by Roboprog on Sunday November 10, 2013 @12:35PM (#6707444)

It sounds to me like the poster above restructured the input data to exploit locality of reference better.

http://en.wikipedia.org/wiki/Locality_of_reference

[ Parent | on HN ]

Re:Ask HN: To everybody who uses MapReduce: what problems do... by collyw on Sunday November 10, 2013 @01:01PM (#6707598)

So assuming the data is one one machine (as I asked), why would an index not solve this problem? And why does Map Reduce solve it?

[ Parent | on HN ]

Re:Ask HN: To everybody who uses MapReduce: what problems do... by virtuabhi on Sunday November 10, 2013 @02:07PM (#6707922)

Indexes do not solve the locality problem (see Non-clustered indexes). Even for in-memory databases, it is non-trivial to minimize cache misses in irregular data structures like B-trees.

Now why MapReduce "might" be a better fit for a problem where data fits into one disk. Consider a program which is embarrassingly parallel. It just reads a tuple and writes a new tuple back to disk. The parallel IO provided by map/reduce can offer a significant benefit in this simple case as well.

Also NoSQL != parallel processing.

[ Parent | on HN ]

Re:Ask HN: To everybody who uses MapReduce: what problems do... by alecco on Sunday November 10, 2013 @12:54PM (#6707563)

It's one of the issues, yes. But I wanted to be more general on purpose.

[ Parent | on HN ]

Re:Ask HN: To everybody who uses MapReduce: what problems do... by alecco on Sunday November 10, 2013 @01:01PM (#6707599)

Maybe I misunderstood what you were asking.

Note both MapReduce and NoSQL are overhyped solutions. They are useful in a handful of cases, but often applied to problems they are not as good.

[ Parent | on HN ]

Re:Ask HN: To everybody who uses MapReduce: what problems do... by aidos on Sunday November 10, 2013 @02:11PM (#6707954)

I'm not sure that the two concepts are resulted at all. Obviously mongo has map reduce baked in - but that's not that relevant. Map/reduce is a reasonable paradigm for crunching information. I have a heavily CPU bound system that I parallelise by running on different machines and aggregating that results. I probably wouldn't call it map reduce - but really it's the same thing.

How do you parallelise your long running tasks otherwise?

[ Parent | on HN ]

Re:Ask HN: To everybody who uses MapReduce: what problems do... by alecco on Sunday November 10, 2013 @02:46PM (#6708090)

I can't say without more information on the problem to solve. As I said above, there are cases where MapReduce is a good tool.

And even if you improve the DS/Algorithm first, usually that is usable by the MapReduce port and you save a lot of time/costs.

[ Parent | on HN ]