Note that the DataFrame version of MLLib (which is still called MLLib btw, even if the package is different) isn't feature complete compared to the RDD version.
For example, the DataFrame version of LogisticRegression is binary only, while the RDD one allows multiple classes. I guess the suggested work around is to use the OneVsRest classifier (in Spark 2.0), but that blows the memory out on fairly small tasks.
I like Spark a lot, but MLLib isn't as strong as it appears at first glance.
Note that the DataFrame version of MLLib (which is still called MLLib btw, even if the package is different) isn't feature complete compared to the RDD version.
For example, the DataFrame version of LogisticRegression is binary only, while the RDD one allows multiple classes. I guess the suggested work around is to use the OneVsRest classifier (in Spark 2.0), but that blows the memory out on fairly small tasks.
I like Spark a lot, but MLLib isn't as strong as it appears at first glance.