Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt...
by sanswork on Saturday June 28, 2014 @01:53AM
(#7957557)
The title has existed for years now at various companies(most with an open source program have them) so whatever you were suppose to do you're too late now.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt...
by j_baker on Saturday June 28, 2014 @01:56AM
(#7957566)
There's a grain of truth to what he's saying. We don't write many new actual C++/Java MapReduce jobs. It's mostly flume. Although there is some momentum behind Go MapReduce.
The key word is pipeline. If you have some analysis that runs in several stages, you'll be taking the output of one stage, and connecting it to the next. If you want to compose multiple phases, chained together, raw MapReduce isn't going to help you very much with the chaining.
What's described in the paper is a way to do the chaining in a nice way. The system will take care of writing the raw MapReduces for you. But it'll also do a lot of work on the interconnections between your stages as well.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt...
by espeed on Saturday June 28, 2014 @01:49AM
(#7957549)
MapReduce wasn't designed for iterative algorithms or streaming data, whereas Google Dataflow and Spark (http://spark.apache.org/) make iterative algoritms easy. It's a much simpler programming paradigm, and it allows you to do iterative graph-processing and machine-learning algos (http://spark.apache.org/mllib/) that are impractical on MapReduce.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt...
by msane on Saturday June 28, 2014 @12:37AM
(#7957434)
Hey sometimes you just have to do whatever it takes to synergize global channels on virtual platforms. You know, really aggregate extensible markets with repurposed leading-edge metrics.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt...
by dclusin on Saturday June 28, 2014 @01:27AM
(#7957519)
But how will these leading-edge metrics enable us to deliver paradigm shifting solutions to our customers while simultaneously reducing costs and increasing operational efficiency?
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt...
by msane on Saturday June 28, 2014 @02:16AM
(#7957591)
That's the best part - it's automated with a context-sensitive customizability infrastructure. That means you can expand any targeted benchmark in a demand-driven mesh.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt...
by jey on Saturday June 28, 2014 @01:26AM
(#7957518)
Cool. Does this mean Google is moving away toward languages that allow for easier use and serialization of closures than in C++ and Java? (For example, Spark uses Scala natively.)
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt...
by jamesaguilar on Saturday June 28, 2014 @12:55AM
(#7957458)
What do you mean, warehousing? Like, item tracking inside an actual warehouse? Hard to imagine spending more than a kB per unique item -- more per SKU, but less per individual object -- so even if you have 1M items being tracked, the total size would only be a gigabyte. Even if you had a billion unique things in your store, the resulting database would still fit on a single flash drive.
This is not even true. They have recently published research that involved using MapReduce in their own systems. Example: http://research.google.com/pubs/pub41376.html
1. Google's infrastructure is evolving atop MapReduce (FlumeJava/MillWheel).
2. Google's PR decided to call it "not using MapReduce anymore" because in marketing, "beyond <current fad>" sounds really cool.
3. The rest of the world's PR/press/marketing fall for Google's clever PR.
Either way, it is great to see Google making its core technology accessible as part of their PaaS =)
Thanks for the clarifications :)
If "hyper-scale Analytics" counts as "clever PR," then eating paste counts as "clever food choice." It's not worth my time to decode the BS.
But what do I do when people start describing themselves as "developer evangelists" ?
That has to be some kind of sell signal, right ?
Citation: http://dl.acm.org/citation.cfm?id=1806638
The key word is pipeline. If you have some analysis that runs in several stages, you'll be taking the output of one stage, and connecting it to the next. If you want to compose multiple phases, chained together, raw MapReduce isn't going to help you very much with the chaining.
What's described in the paper is a way to do the chaining in a nice way. The system will take care of writing the raw MapReduces for you. But it'll also do a lot of work on the interconnections between your stages as well.
For example, Spark provides the primitives needed to build GraphX (http://amplab.github.io/graphx/, http://spark.apache.org/graphx/), which is essentially GraphLab on Spark.
Buzzword... overload!
Here are the relevant papers...
* FlumeJava (iterative, data-parallel pipelines like Spark): http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/F...
* MillWheel (fault-tolerant stream processing like SparkStreaming): http://research.google.com/pubs/pub41378.html
Pointers to the IO blog posts...
* "Reimagining developer productivity and data analytics in the cloud" http://googlecloudplatform.blogspot.com/2014/06/reimagining-...
* "Sneak peek: Google Cloud Dataflow, a Cloud-native data processing service" http://googlecloudplatform.blogspot.com/2014/06/sneak-peek-g...
The Dataflow-specific talks at Google IO 2014...
* Big data, the Cloud way: Accelerated and simplified https://www.youtube.com/watch?v=Y0Z58YQSXv0
* The dawn of "Fast Data" https://www.youtube.com/watch?v=TnLiEWglqHk
* Predicting the future with the Google Cloud Platform https://www.youtube.com/watch?v=YyvvxFeADh8
* Keynote (starts at Urs Hölzle's segment on Google Cloud) https://www.youtube.com/watch?v=wtLJPvx7-ys#t=6932
The rest is buzzwords propping up sweeping ridiculous conclusions.
I dont think there's a lot of companies where data would reach this huge. Anyone has any idea on how large a typical warehousing database is?
While I couldn't quickly find anything that speaks to any kind of average or normal size of a data warehouse, this article mentions Facebook's being around 300PB: https://code.facebook.com/posts/229861827208629/scaling-the-...
Seriously.