Re:In Media, Big Data Is Booming but Big Results Are Lacking
by izendejas on Saturday May 25, 2013 @11:33PM
(#5769921)
You should look into Spark: http://spark-project.org/ and Shark (Spark + Hive).
I personally find that using the Spark repl is both efficient and flexible; and being able to cache data in memory makes for very fast iterations, especially when doing something more sophisticated.
There's also Cloudera Impala, but I've never used it, nor do I think I'll ever have to given what the AMPLab folks are doing.
The way "big data" is stored today also poses problems with respect to queryability. NoSQL systems are great at storing huge amounts of data efficiently but don't help us slice and dice the data easily. Having to write map reduce jobs for every "query" is painful and time consuming. Tools like Hive, Pig and Cascading help in writing MR jobs succinctly but are still very slow when someone wants to quickly filter and explore the data.