News for hackers. Stuff that matters.
Re:Spark Breaks Previous Large-Scale Sort Record by AustinBGibbons on Friday October 10, 2014 @01:17PM (#8438927)
Hi Reynold! Do you have numbers / intuition for how previous versions of spark would have run? I'm upgrading (soon) from spark 0.8 to spark 1.1 and am curious to see the performance gains (especially w.r.t. shuffles)
Re:Spark Breaks Previous Large-Scale Sort Record by rxin on Friday October 10, 2014 @02:07PM (#8439193)
Hi Austin,

We haven't tested Spark 0.8 at this scale. In general Spark is advancing at a rapid rate that 1.1 is very very different from 0.8.