Don't use Hadoop when your data isn't that big

Re:Don't use Hadoop when your data isn't that big by sitkack on Tuesday September 17, 2013 @06:14PM (#6402277)

Hadoop is the problem, MapReduce is not the problem. Having used both Hadoop and Disco, I can say that Disco is by far a net positive on all projects I used it on. And the overhead to coding it in Disco vs single node is about an extra 30 minutes. You can start with working single node and go multinode w/o much effort.

http://discoproject.org

Hadoop on the other hand is a huge, massive pain in the ass. And I am a Hadoop consultant. I recommend that most customers NOT use it.

[ Parent | on HN ]

Re:Don't use Hadoop when your data isn't that big by travisoliphant on Wednesday September 18, 2013 @04:14AM (#6404125)

This is 100% true. Hadoop gets way too much attention given the other useful solutions that exist out there. I have known people to use Disco successfully on several hundred node cluster. You can also interact Disco with IPython parallel much more easily.

This is why we include it in Linux versions of Anaconda.

[ Parent | on HN ]

Re:Don't use Hadoop when your data isn't that big by hhuuggoo on Wednesday September 18, 2013 @09:14AM (#6405014)

agreed, disco is really great becuase you can easily get access to the file descriptor which has your data, then you can load it however way you want

[ Parent | on HN ]

Re:Don't use Hadoop when your data isn't that big by edraferi on Tuesday September 17, 2013 @11:39PM (#6403311)

Cool project, thanks for the link.

[ Parent | on HN ]