News for hackers. Stuff that matters.
Re:Sorting a Billion Numbers with Julia by rxin on Tuesday March 22, 2016 @01:28AM (#11334588)
As part of Spark 2.0, we are introducing some new neat optimizations to make a general engine as efficient as specialized code.

I just tried on Spark master branch (i.e. the work-in-progress code for Spark 2.0). It takes about 1.5 secs to sum up 1 billion 64-bit integers using a single thread, and about 1 secs using 2 threads. This was done on my laptop (Early 2015 Macbook Pro 13, 3.1GHz Intel Core i7).

We haven't optimized integer sorting yet, so that's probably not going to be super fast, but the aggregation performance has been pretty good.

  scala> val start = System.nanoTime
  start: Long = 56832659265590
  scala> sqlContext.range(0, 1000L * 1000 * 1000, 1, 2).count()
  res8: Long = 1000000000
  scala> val end = System.nanoTime
  end: Long = 56833605100948
  scala> (end - start) / 1000 / 1000
  res9: Long = 945
Part of the time are actually spent analyzing the query plan, optimizing it, and generating bytecode for it. If we run this on 10 billion integers, the time is about 5 secs.

There are no comments on this comment yet.