Setting a new world record in CloudSort with Apache Spark

Re:Setting a new world record in CloudSort with Apache Spark by embiggen on Tuesday November 15, 2016 @05:52PM (#12963061)

Meanwhile, google was sorting Petabytes in under a minute on their clusters 6+ years ago. We've still got a long ways to go in OSS land to compete with the big boys.

[ Parent | on HN ]

Re:Setting a new world record in CloudSort with Apache Spark by fhoffa on Tuesday November 15, 2016 @07:05PM (#12963497)

This post tells the "History of massive-scale sorting experiments at Google"

- https://cloud.google.com/blog/big-data/2016/02/history-of-ma...

When I asked why BigQuery doesn't do these sorts, the answer came straight from the post "Nobody really wants a huge globally-sorted output. We haven’t found a single use case for the problem as stated."

These accomplishments are awesome nevertheless!

Disclaimer: I'm Felipe Hoffa, and I work for Google (http://twitter.com/felipehoffa).

[ Parent | on HN ]

Re:Setting a new world record in CloudSort with Apache Spark by andrioni on Tuesday November 15, 2016 @07:57PM (#12963727)

Do you think you could ask someone and find out the cluster sizes they used for those sorts? They mention "With the largest cluster at Google under our control", but it would be more interesting to have an idea of actual numbers, even if just an order of magnitude.

[ Parent | on HN ]

Re:Setting a new world record in CloudSort with Apache Spark by fhoffa on Tuesday November 15, 2016 @08:26PM (#12963847)

I could ask - but then I wouldn't be able to publish unpublished numbers on my own (if I want to keep my job).

[ Parent | on HN ]