A New Analytics Toolbox with Apache Spark – Going Beyond Hadoop

Re:A New Analytics Toolbox with Apache Spark – Going Beyond ... by x0x0 on Tuesday August 26, 2014 @11:49PM (#8230787)

In practice -- again, see my experience writing jobs processing pbs of data on 1000s of nodes -- hadoop experiences namenode issues. Hadoop was designed many ways, blah blah blah, but that's essentially not what the software is as shipped. And that's about it; I was going to respond point by point, but your post displays a general lack of knowledge of large-scale hadoop deployments, so it's basically a waste of time.

[ Parent | on HN ]

Re:A New Analytics Toolbox with Apache Spark – Going Beyond ... by colin_mccabe on Wednesday August 27, 2014 @12:07AM (#8230823)

I work for a big Hadop vendor and I regularly diagnose performance issues with big clusters. Yes, sometimes even thousands of nodes. Most of the time, it comes down to an incorrect assumption or configuration. There are certainly real bugs and shortcomings, but the bottom line is that the systems we sell are orders of magnitude cheaper to build and operate than what they replace.

And that's about it; I was going to respond point by point, but your post displays a general lack of knowledge of large-scale hadoop deployments, so it's basically a waste of time.

You can't even respond to a single point? I'm sorry to hear that. I have been polite and respectful to you, can you pay me the same courtesy?

[ Parent | on HN ]

Re:A New Analytics Toolbox with Apache Spark – Going Beyond ... by cbsmith on Wednesday August 27, 2014 @12:33AM (#8230865)

You are playing from the old playbook though. Heck, we have HA NameNode setups and name node clusters now, and some implementations of Hadoop don't even use HDFS. The Hadoop's tech stack has changed a lot over the years. It's much different now. It's worth noting that there are many examples now of clusters with five figures worth of nodes.

[ Parent | on HN ]