I did hit issues w/ multiple joins and shuffling though. Have you not hit issues w/ shuffling?
I was using Spark 1.5.1 for the record.
That is a good metric for a project.
But let's be clear here Spark is not a 'real' open source project. It's a dictatorship run by Databricks (in the nicest possible way). There are 414 outstanding pull requests some of which came from the likes of Intel which added subquery support over a year ago. Never got merged.
The best metric for any open source project is how easily can you contribute and improve the product.
I showed up on the mailing list, said "hey, here's some stuff that was useful for me, let me know if it's useful for anyone else." A committer said, "cool, send a PR". Couple of rounds of code review and it was merged. Later on, bigger features required a lot more discussion, but as long as I was willing to follow up, I got my changes in.
Spark has on the order of 1,000 contributors. I'd call that a "real" open source project.
I think a prime example of this is I was using some very basic windowing functions and due to the data shuffling (The data wasn't naturally partitioned) it seemed to be very buggy and not very clear why stuff was failing. I ended up rewriting the same section of code using hive and it had both better performance and didn't seem to have any odd failures. I realize this stuff will improve but I'm still skeptical.