News for hackers. Stuff that matters.
Re:Spark as a Compiler: Joining a Billion Rows per Second on... by tma-1 on Monday May 23, 2016 @01:53PM (#11755547)
I have been extensively using the dateframe/sql API and I just love it. Most of the issues I have had stemmed from the cluster / Spark configuration and not the API itself. Using SQL is so much more intuitive them using multiple joins, selects, filter etc on an rdd.
Re:Spark as a Compiler: Joining a Billion Rows per Second on... by mastratton3 on Monday May 23, 2016 @05:53PM (#11757361)
So I did find it useful for doing additional exploratory aggregations once the data was already cleaned and denormalized. My comment was more directed at the upfront initial data processing (In our case, extracting time series data out of a large amount of files).

I did hit issues w/ multiple joins and shuffling though. Have you not hit issues w/ shuffling?

I was using Spark 1.5.1 for the record.

Re:Spark as a Compiler: Joining a Billion Rows per Second on... by tma-1 on Monday May 23, 2016 @08:14PM (#11758002)
Have you tried tuning Spark's memory parameters?