Setting a new world record in CloudSort with Apache Spark

Re:Setting a new world record in CloudSort with Apache Spark by devonkim on Tuesday November 15, 2016 @01:50PM (#12960981)

It's not clear how much of these improvements are from reductions in pricing rather than algorithms and design decisions. They've documented things like using Netty for network latency, avoiding GC, and getting better with Spark, but it'd be interesting if the team could go back and run the benchmark using the same infrastructure as their 2014 benchmark for a code-vs-code comparison to separate engineering improvements from economies of scale.

[ Parent | on HN ]

Re:Setting a new world record in CloudSort with Apache Spark by rxin on Tuesday November 15, 2016 @02:14PM (#12961198)

I definitely agree that it'd be great to decouple the improvements in software and drops in cloud pricing. In reality it is pretty difficult because the Nanjing U/Alibaba team spent a lot of time also optimizing the software specifically for the AliCloud environment, which might not be applicable when running on Amazon EC2, which was the environment of the 2014 record.

This is a great task for a rigorous academic paper!

Disclaimer: I wrote the blog post.

[ Parent | on HN ]

Re:Setting a new world record in CloudSort with Apache Spark by wsmith on Tuesday November 15, 2016 @03:21PM (#12961827)

Is Spark faster than MemSQL?

[ Parent | on HN ]

Re:Setting a new world record in CloudSort with Apache Spark by gopalv on Tuesday November 15, 2016 @04:37PM (#12962461)

MemSQL is a transactional database (system of record).

Spark is a way of processing data, ideally stored in a system of record (Hive/HDFS/S3/MemSQL etc).

They're not the same.

[ Parent | on HN ]

Re:Setting a new world record in CloudSort with Apache Spark by wsmith on Wednesday November 16, 2016 @08:05AM (#12966528)

There are similarities. A database is also a way of processing data.

For the kinds of processing both Spark and MemSQL do (e.g. join operation) is Spark faster than MemSQL?

[ Parent | on HN ]

Re:Setting a new world record in CloudSort with Apache Spark by Bedon292 on Tuesday November 15, 2016 @06:53PM (#12963427)

I agree, hard to understand the changes. Actually fond some addition info as I was writing this though. http://sortbenchmark.org/

Old record was 330 r3.4xlarge machines. Which actually cost very similar today. It looks like the old record used a LOT more RAM than this one. 40,260GB vs 3,152GB and similar CPU 5,280 vs 4,728 cores. Although having the RAM doesn't mean it was used, but unless they were using all the RAM I don't see why they would use the more expensive r3 instances.

Edit: Follow on, I think the cost savings is definitely in the low RAM usage. Can't really get that many cores without more than 3 times as much RAM in AWS.

[ Parent | on HN ]