It's true that PySpark probably is overtaking (or already has overtaken) Spark with Scala. I still personally like Scala over Java in general for non-Spark projects as it's much less verbose which is my main problem with Java. However, I also haven't used all the new Java features to see how that would adjust the comparison for me.