News for hackers. Stuff that matters.
Re:MapReduce and Spark by lmm on Thursday January 02, 2014 @07:36PM (#7003500)
I like Spark over Hadoop just from an interface point of view, particularly the ability to just start up a (Scala) shell and start playing around. Hadoop can be very effective, but even getting "hello world" to run requires an intimidating array of setup.
Re:MapReduce and Spark by rxin on Thursday January 02, 2014 @07:57PM (#7003574)
There is also a Python shell now :)
Re:MapReduce and Spark by gknight on Thursday January 02, 2014 @08:47PM (#7003724)
Have you tried Apache Hive? I believe it was meant to make Hadoop easier to use by way of SQL-like commands. Something like Qubole might be able to help too.
Re:MapReduce and Spark by lmm on Friday January 03, 2014 @05:58AM (#7004879)
I have, I should've mentioned that as an option. But I find it much easier to think in Scala than in an SQL-like language.
Re:MapReduce and Spark by pavanred on Thursday January 02, 2014 @10:07PM (#7003918)
Pig is another option. It allows using SQL like commands on the grunt shell, making using Hadoop a lot easier.
Re:MapReduce and Spark by rxin on Thursday January 02, 2014 @10:27PM (#7003973)
Going from Hive/Pig to Spark enables substantial improvement in developers' productivity (for non-reporting/BI workloads). You can properly unit test your program, use a debugger, and have all your code in the same place in the same language (rather than in the case of Pig, write UDFs in Java and then use a pseudo-scripting language for workflow specification).

All of these are just productivity gains; not to mention the performance gains you get when you go from MapReduce to Spark.

Re:MapReduce and Spark by ianburrell on Thursday January 02, 2014 @10:39PM (#7004008)
There is Shark which is Hive implemented on top of Spark instead of Hadoop.