News for hackers. Stuff that matters.
Re:Some Interesting Facts About Hadoop MapReduce [pdf] by shuzchen on Wednesday February 27, 2013 @06:20PM (#5295754)
I'm not a "power-user", but I'm in the process of building a pretty big product powered by hadoop (more specifically, mrjob and amazon's EMR).

I have a hard time reading this as anything but satire. The author definitely knows a lot about hadoop, yet displays a misunderstanding of such a basic piece of it (conflating hadoop with map-reduce).

Just a few key things I can refute:

Hadoop as wasteful: production systems are built to withstand the expected peaks of traffic (i.e., some website that sees a huge influx when the east coast wakes up on monday morning will design with that load+buffer in mind), which means that ~90% (probably even higher!) of the time resources are underutilized. even personally my own laptop isn't chugging at 100% on all cores all day every day, I don't see how anybody expects any collection of computers to do so. On the flip side, this is being offset by platforms like AWS. When I run map-reduce jobs on hadoop, I spin up an EMR jobflow, run a few tasks and then shut it down. Presumably, somebody else gets to use those machines after I'm done.

Lack of third party support: Honestly, with mrjob and EMR, I've never needed to deal with any hadoop management, setup, maintenance or scaling. Also, all my code is python, I've not had to write a line of Java. And these aren't the only tools built on top of hadoop.

Hadoop as deprecated: there are certainly more big-data crunching platforms coming out nowadays, but hadoop is mature, stable, and still going strong. Also, the basic map-reduce framework it's based on is not going anywhere soon. I mean, I cut my teeth on map-reduce with mongodb, and it seems most other nosql solutions there are going that route for complicated things their standard query engine can't support.

There are no comments on this comment yet.