News for hackers. Stuff that matters.
Re:A New Analytics Toolbox with Apache Spark – Going Beyond ... by srean on Tuesday August 26, 2014 @06:24PM (#8229917)
I already upvoted, but this needs more, thanks for the excellent read. Some of the inefficiencies you describe are pretty crazy ! If any of the optimizations you did are documented in blogs, would love to have a pointer.

@cbsmith Just in case you have links handy could you post them. Much appreciated.

Re:A New Analytics Toolbox with Apache Spark – Going Beyond ... by cbsmith on Tuesday August 26, 2014 @06:44PM (#8229994)
These issues with Hadoop and HDFS in particular are pretty well documented even by people in the Hadoop project. There was also a bunch of blog posts posted at Yahoo shortly after Cutting's departure that looked at a bunch of other issues with the IO subsystem.

Much of this stuff has been heavily refactored, particularly with Hadoop 2.0, so the problems are at least much smaller than they used to be. Early Hadoop was not exactly a masterful work in terms of the code & algos. What was great about it was the community that was built around it.

Re:A New Analytics Toolbox with Apache Spark – Going Beyond ... by x0x0 on Tuesday August 26, 2014 @09:05PM (#8230415)
see eg quantcast's experiences with why hadoop is slow on big jobs [1], the design of their sorter architecture [2], and some benchmark wanking [3]

[1] https://www.quantcast.com/inside-quantcast/2012/10/letter-fr...

[2] https://www.quantcast.com/inside-quantcast/2012/10/quantsort...

[3] https://www.quantcast.com/inside-quantcast/2013/12/petabyte-...