Re:Announcing Spark 1.6
by mziel on Monday January 04, 2016 @04:00PM
(#10838559)
To be honest PySpark and SparkR are always going to be 2nd category citizens (because of the serialization/pickling between the two environments). Databricks shows nice graphs, saying they are equivalent for DataFrames, however those count only for built-in functions that basically translate code into execution plan for Catalyst. For anything bespoke (UDFs, custom Transformers/Estimators) you're better off using Scala.
[ Parent
| on HN ]
This is true when you compare the performance vs Java/Scala, but if you compare it with other tools that are native in Python, it is not really much worse. For examples, Pandas operations that use custom UDFs are substantially slower than the native operations.
That said, as part of Project Tungsten, we have some ideas about a batch columnar format that can be shared by Python, R, Scala and Java, and that should be able to eliminate most of the inefficiency in serialization across process boundaries.
That sounds very interesting. Is there any ticket, where I can follow the progress on the batch columnar format you mentioned?
Btw, I was critical about the issue above, but I do love Spark, using it on a daily basis. :)
I just created a JIRA ticket tracking this: https://issues.apache.org/jira/browse/SPARK-12635
Thanks for the reminder!
Yes that's right. Team: could you make more API available through PySPark, please?
That's fair; mostly I'd prefer keeping everything in Python due to the other data implications outside of Spark (e.g bespoke data cleanup with Python syntax) and I'd prefer to keep the stack smaller.