Command-line tools can be faster than your Hadoop cluster

Re:Command-line tools can be faster than your Hadoop cluster by dalke on Sunday January 18, 2015 @03:42PM (#8908792)

Indeed. For more real-world examples of people who thought that they had "big data", but didn't, see https://news.ycombinator.com/item?id=6398650 ("Don't use Hadoop - your data isn't that big"). The linked essay has:

> They handed me a flash drive with all 600MB of their data on it (not a sample, everything). For reasons I can't understand, they were unhappy when my solution involved pandas.read_csv rather than Hadoop.

User w_t_payne commented:

> I have worked for at least 3 different employers that claimed to be using "Big Data". Only one of them was really telling the truth.

> All of them wanted to feel like they were doing something special.

I think that last line is critical to understanding why a CIO might feel this way.

[ Parent | on HN ]

Re:Command-line tools can be faster than your Hadoop cluster by nathancahill on Sunday January 18, 2015 @05:30PM (#8909224)

"Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it." -Dan Ariely

[ Parent | on HN ]