Setting a new world record in CloudSort with Apache Spark

Re:Setting a new world record in CloudSort with Apache Spark by rxin on Tuesday November 15, 2016 @02:14PM (#12961198)

I definitely agree that it'd be great to decouple the improvements in software and drops in cloud pricing. In reality it is pretty difficult because the Nanjing U/Alibaba team spent a lot of time also optimizing the software specifically for the AliCloud environment, which might not be applicable when running on Amazon EC2, which was the environment of the 2014 record.

This is a great task for a rigorous academic paper!

Disclaimer: I wrote the blog post.

[ Parent | on HN ]

Re:Setting a new world record in CloudSort with Apache Spark by wsmith on Tuesday November 15, 2016 @03:21PM (#12961827)

Is Spark faster than MemSQL?

[ Parent | on HN ]

Re:Setting a new world record in CloudSort with Apache Spark by gopalv on Tuesday November 15, 2016 @04:37PM (#12962461)

MemSQL is a transactional database (system of record).

Spark is a way of processing data, ideally stored in a system of record (Hive/HDFS/S3/MemSQL etc).

They're not the same.

[ Parent | on HN ]

Re:Setting a new world record in CloudSort with Apache Spark by wsmith on Wednesday November 16, 2016 @08:05AM (#12966528)

There are similarities. A database is also a way of processing data.

For the kinds of processing both Spark and MemSQL do (e.g. join operation) is Spark faster than MemSQL?

[ Parent | on HN ]