News for hackers. Stuff that matters.
Re:Spark Breaks Previous Large-Scale Sort Record by rxin on Friday October 10, 2014 @12:41PM (#8438695)
Thanks for sharing this. I'm the author of this blog post. Free free to ask me anything.
Re:Spark Breaks Previous Large-Scale Sort Record by chad_walters on Friday October 10, 2014 @01:41PM (#8439057)
Your post mentions "single root IO virtualization" as a factor in maximizing network performance. I am wondering what the impact of this was in your sorting. Do you have data for runs where you didn't enable this?
Re:Spark Breaks Previous Large-Scale Sort Record by rxin on Friday October 10, 2014 @02:05PM (#8439182)
It was part of the enhanced networking. Without enhanced networking, we were getting about 600MB/s, vs 1.1GB/s with.
Re:Spark Breaks Previous Large-Scale Sort Record by AustinBGibbons on Friday October 10, 2014 @01:17PM (#8438927)
Hi Reynold! Do you have numbers / intuition for how previous versions of spark would have run? I'm upgrading (soon) from spark 0.8 to spark 1.1 and am curious to see the performance gains (especially w.r.t. shuffles)
Re:Spark Breaks Previous Large-Scale Sort Record by rxin on Friday October 10, 2014 @02:07PM (#8439193)
Hi Austin,

We haven't tested Spark 0.8 at this scale. In general Spark is advancing at a rapid rate that 1.1 is very very different from 0.8.

Re:Spark Breaks Previous Large-Scale Sort Record by ch on Friday October 10, 2014 @03:09PM (#8439498)
Curious if using the Sparrow scheduler would have been a net gain/loss to this type of work load?
Re:Spark Breaks Previous Large-Scale Sort Record by rxin on Friday October 10, 2014 @03:33PM (#8439603)
It would help a little bit (maybe a few percent), but not much because the scheduling latency was relatively low for these tasks (the largest scheduling delay was ~10 secs, whereas each task takes minutes).