Re:A New Analytics Toolbox with Apache Spark – Going Beyond ...
by cbsmith on Tuesday August 26, 2014 @06:44PM
(#8229994)
These issues with Hadoop and HDFS in particular are pretty well documented even by people in the Hadoop project. There was also a bunch of blog posts posted at Yahoo shortly after Cutting's departure that looked at a bunch of other issues with the IO subsystem.
Much of this stuff has been heavily refactored, particularly with Hadoop 2.0, so the problems are at least much smaller than they used to be. Early Hadoop was not exactly a masterful work in terms of the code & algos. What was great about it was the community that was built around it.
Re:A New Analytics Toolbox with Apache Spark – Going Beyond ...
by x0x0 on Tuesday August 26, 2014 @09:05PM
(#8230415)
see eg quantcast's experiences with why hadoop is slow on big jobs [1], the design of their sorter architecture [2], and some benchmark wanking [3]
[1] https://www.quantcast.com/inside-quantcast/2012/10/letter-fr...
[2] https://www.quantcast.com/inside-quantcast/2012/10/quantsort...
[3] https://www.quantcast.com/inside-quantcast/2013/12/petabyte-...
@cbsmith Just in case you have links handy could you post them. Much appreciated.