Admittedly I'm not a spark expert though, thanks for bringing this up!
RDDs themselves, however, are obsolete to DataFrames as DataFrames are faster and cover most of the functionality of RDDs for common use cases.
So if you want to do one of the operations in the sql.functions package[1] then Dataframes (and Datasets) are very valuable.
If not, then they won't give you much benefit. However, you will get a little improvement because the Tachyon out-of-JVM-memory framework[2] which I don't think RDD version has access to.
[1] http://spark.apache.org/docs/2.0.0/api/python/pyspark.sql.ht...
[2] https://dzone.com/articles/Accelerate-In-Memory-Processing-w...
Pushdown is the most obvious one. If I don't know what data store is underlying your RDD, I don't know your schema, and I don't know what column you're projecting, pushdown is impossible. I can't know that with an RDD, because all I know when you call map is that you're converting from type A to type B.
Dataframes make that class of optimization possible, because they have more information (your schema, the underlying store), and have more limited operations (select a column, not an arbitrary map operation).
ALS has been available in ML/DataFrames since 1.3.0, per the documentation. (http://spark.apache.org/docs/latest/api/scala/index.html#org...)
> Edit: Looks like the original comment was edited, but this post does in fact use the built in MLlib cross validation methods.
I edited the comment to correct my error that ALS was not mentioned as being native to Spark. However, for Hyperparameter cross validation, I looked at the code in the repository (https://github.com/sigopt/sigopt-examples/blob/master/spark/...), and while the Spark CrossValidation function is imported at the beginning of the Scala file (from ML, not MLlib), it is not used in the code, in favor of SigOpt.
I'm confused.