News for hackers. Stuff that matters.
Re:Spark: Open Source Superstar Rewrites Future of Big Data by DiabloD3 on Thursday June 20, 2013 @12:33AM (#5910164)
I don't understand why they wrote this in Java instead of a language more suited to the task, such as Erlang.

Can someone explain this to me?

Re:Spark: Open Source Superstar Rewrites Future of Big Data by penland on Thursday June 20, 2013 @01:00AM (#5910266)
According to github, 85% of this is in Scala.

It's written on the JVM for the simple reason that if they wrote it in Go or Erlang, no Enterprise would adopt it as there isn't a CTO at a non-tech Fortune 500 that has every heard of Erlang or GO, and wouldn't know the first thing about trying to hire developers for it. Remember, the jobs written for Map Reduce are done in the same language ( typically ) as the MapReduce code itself.

Re:Spark: Open Source Superstar Rewrites Future of Big Data by shin_lao on Thursday June 20, 2013 @05:55AM (#5911126)
Why didn't they go native?
Re:Spark: Open Source Superstar Rewrites Future of Big Data by pkolaczk on Thursday June 20, 2013 @07:37AM (#5911438)
Why would they?
Re:Spark: Open Source Superstar Rewrites Future of Big Data by shin_lao on Thursday June 20, 2013 @07:49AM (#5911486)
I thought they cared about performance...
Re:Spark: Open Source Superstar Rewrites Future of Big Data by pkolaczk on Thursday June 20, 2013 @01:30AM (#5910359)
It is written in Scala with Akka framework, not pure Java. So it uses the same computational model as Erlang. Thanks to being Java-compatible it is easier to integrate with some other popular BigData tools, e.g. Hadoop or Hive (see Shark).
Re:Spark: Open Source Superstar Rewrites Future of Big Data by reeses on Thursday June 20, 2013 @10:59PM (#5916271)
Erlang is not that performant in the general case compared to C/Java/etc. While the conceptual overhead in distribution is much lighter with Erlang, actual computation in an idiomatic implementation tends to be slower than Scala/Akka.

Of course, there are tasks for which an Erlang implementation would be faster, but as others have mentioned, most organizations would prefer to write Scala.

(Yes, I find that icky too. :-))

Re:Spark: Open Source Superstar Rewrites Future of Big Data by fixxer on Thursday June 20, 2013 @12:38AM (#5910182)
More than a few other choices (Go...), not to mention alternatives still on the JVM.

Nobody ever got fired for building with Java.

Re:Spark: Open Source Superstar Rewrites Future of Big Data by pkolaczk on Thursday June 20, 2013 @07:43AM (#5911461)
Go is nowhere near the adoption of Scala right now, and its compiler must catch up a lot to get to the similar level of performance as Scala has on top of Oracle JVM.

Considering there are already lots of BigData tools in Java ecosystem (hadoop, hive, pig, mahout etc.) Scala looks like a very reasonable choice.

Re:Spark: Open Source Superstar Rewrites Future of Big Data by fixxer on Thursday June 20, 2013 @11:33PM (#5916371)
You statement is perfectly valid. Scala was in my mind when I wrote my comment.

I wish I found Scala as easy to use as I do Go. I do not enjoy the syntax, nor do I consider running on top of the JVM a selling point.

I came from a Java, but worked in Python for many years, so that probably explains my bias.