News for hackers. Stuff that matters.
Re:Announcing Apache Spark 1.4 by minimaxir on Thursday June 11, 2015 @02:55PM (#9701551)
I'm excited about SparkR, even though R is shunned in the field of big data. Between that and dplyr (which inspired the SparkR syntax) for data manipulation and sanitation, it should be much easier to write sane, reproducible code and visualizations for big data analysis. (the Python/Scala tutorials for Spark gave me a headache)

SparkR appears to have strong integration into Rstudio, which is big news: http://blog.rstudio.org/2015/05/28/sparkr-preview-by-vincent...

Re:Announcing Apache Spark 1.4 by threeseed on Thursday June 11, 2015 @10:11PM (#9703629)
R is absolutely not shunned in big data. It is very popular.

There is a reason Microsoft acquired Revolution Analytics.

Re:Announcing Apache Spark 1.4 by eranation on Thursday June 11, 2015 @06:09PM (#9702773)
R on Spark is great, but the biggest issue in my view is R's runtime licensing, isn't it GPL? Am I worried for nothing?
Re:Announcing Apache Spark 1.4 by zmmmmm on Thursday June 11, 2015 @07:51PM (#9703175)
I too have been mystified by R's licensing. I actually don't see how anyone can ship a commercial product using R in its current form. At very least you're in a legal gray area, at worst you are involuntarily open sourcing your product. Not that there's anything wrong with open sourcing a product, but I think there's an enormous potential issue that could foul up a lot of people down the track. The best discussion I have seen about this pretty much ends up with uninformed speculation. For now, I take the policy of "explore and prototype in R, build the real system in something else". Fortunately the flaws and limitations of R as a language make this a sensible choice for a host of other reasons as well.
Re:Announcing Apache Spark 1.4 by mwexler on Thursday June 11, 2015 @05:17PM (#9702511)
Not sure that R is _shunned_ in big data, as much as there are better solutions once you get to a certain level of big.
Re:Announcing Apache Spark 1.4 by IndianAstronaut on Thursday June 11, 2015 @03:37PM (#9701814)
It will be interesting to see how all the R libraries play with Spark. There are bound to be some hiccups there.
Re:Announcing Apache Spark 1.4 by minimaxir on Thursday June 11, 2015 @04:07PM (#9702008)
My interpretation is that it will convert DataFrames to normal data.frames when necessary. Unfortunately, this removes the performance efficiency of Spark.

Since currently SparkR only supports aggregation, it limits the usability of SparkR slightly. Future versions will apparently have MLib support which should alleviate that.