Re:Microsoft announces major commitment to Apache Spark
by nchammas on Monday June 06, 2016 @03:31PM
(#11849605)
You're right about Python and R having to pass data back and forth to the JVM for certain operations, but also keep in mind that native code still runs in the native interpreter. That means you have access to the full ecosystem of the native language.
For example, if I want to convert an RDD of JSON strings into Python dictionaries:
Same goes for any external Python libraries I install on the cluster and want to use in my Spark job. You can even run your Python code on PyPy [4]!
For me, working in Python generally feels like a first class experience on Spark. There are areas -- like GraphX [0], certain niche features [1] -- where Scala is definitely easier to work with, but with time that is becoming less [2] and less [3] true thanks to the DataFrame API.
For example, if I want to convert an RDD of JSON strings into Python dictionaries:
Same goes for any external Python libraries I install on the cluster and want to use in my Spark job. You can even run your Python code on PyPy [4]!For me, working in Python generally feels like a first class experience on Spark. There are areas -- like GraphX [0], certain niche features [1] -- where Scala is definitely easier to work with, but with time that is becoming less [2] and less [3] true thanks to the DataFrame API.
[0] https://spark.apache.org/graphx/
[1] http://stackoverflow.com/q/23995040/877069
[2] https://github.com/graphframes/graphframes
[3] http://stackoverflow.com/a/37150604/877069
[4] https://github.com/apache/spark/pull/2144