News for hackers. Stuff that matters.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by _dark_matter_ on Saturday June 28, 2014 @12:29AM (#7957420)
>“We don’t really use MapReduce anymore”

This is not even true. They have recently published research that involved using MapReduce in their own systems. Example: http://research.google.com/pubs/pub41376.html

Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by kiyoto on Saturday June 28, 2014 @12:44AM (#7957442)
As someone who has one foot in marketing and one foot in development (I am a developer evangelist), here is what's happening.

1. Google's infrastructure is evolving atop MapReduce (FlumeJava/MillWheel).

2. Google's PR decided to call it "not using MapReduce anymore" because in marketing, "beyond <current fad>" sounds really cool.

3. The rest of the world's PR/press/marketing fall for Google's clever PR.

Either way, it is great to see Google making its core technology accessible as part of their PaaS =)

Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by zobzu on Saturday June 28, 2014 @01:05AM (#7957481)
hyperscale man! cyberleetscale would have been better. IMO.

Thanks for the clarifications :)

Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by username223 on Saturday June 28, 2014 @01:06AM (#7957483)
> 3. The rest of the world's PR/press/marketing fall for Google's clever PR.

If "hyper-scale Analytics" counts as "clever PR," then eating paste counts as "clever food choice." It's not worth my time to decode the BS.

Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by t1m on Saturday June 28, 2014 @12:59AM (#7957466)
It would be more great if they open sourced it.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by rsync on Saturday June 28, 2014 @01:23AM (#7957511)
I know what I'm supposed to do when my taxi driver starts recommending stocks to me ...

But what do I do when people start describing themselves as "developer evangelists" ?

That has to be some kind of sell signal, right ?

Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by sanswork on Saturday June 28, 2014 @01:53AM (#7957557)
The title has existed for years now at various companies(most with an open source program have them) so whatever you were suppose to do you're too late now.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by alphapapa on Saturday June 28, 2014 @01:43AM (#7957542)
Looks like someone thinks you're asking too many questions. :/
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by j_baker on Saturday June 28, 2014 @01:56AM (#7957566)
There's a grain of truth to what he's saying. We don't write many new actual C++/Java MapReduce jobs. It's mostly flume. Although there is some momentum behind Go MapReduce.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by basyt on Saturday June 28, 2014 @12:32AM (#7957425)
Its fundamentally the same thing as MapReduce isn't it? Can someone explain the differences to me please? There isn't much of use in the article
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by dyoo1979 on Saturday June 28, 2014 @01:42AM (#7957541)
You'll probably want to read the FlumeJava paper. http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/F...

Citation: http://dl.acm.org/citation.cfm?id=1806638

The key word is pipeline. If you have some analysis that runs in several stages, you'll be taking the output of one stage, and connecting it to the next. If you want to compose multiple phases, chained together, raw MapReduce isn't going to help you very much with the chaining.

What's described in the paper is a way to do the chaining in a nice way. The system will take care of writing the raw MapReduces for you. But it'll also do a lot of work on the interconnections between your stages as well.

Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by espeed on Saturday June 28, 2014 @01:49AM (#7957549)
MapReduce wasn't designed for iterative algorithms or streaming data, whereas Google Dataflow and Spark (http://spark.apache.org/) make iterative algoritms easy. It's a much simpler programming paradigm, and it allows you to do iterative graph-processing and machine-learning algos (http://spark.apache.org/mllib/) that are impractical on MapReduce.

For example, Spark provides the primitives needed to build GraphX (http://amplab.github.io/graphx/, http://spark.apache.org/graphx/), which is essentially GraphLab on Spark.

Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by njharman on Saturday June 28, 2014 @01:05AM (#7957480)
This has "cloud" prefixed to name of every component. So, obviously, is better. Also, they're selling it. So, ya know, marketing trumps engineering.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by wyager on Saturday June 28, 2014 @12:03AM (#7957366)
>Google Replaces MapReduce With New Hyper-Scale Cloud Analytics System

Buzzword... overload!

Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by msane on Saturday June 28, 2014 @12:37AM (#7957434)
Hey sometimes you just have to do whatever it takes to synergize global channels on virtual platforms. You know, really aggregate extensible markets with repurposed leading-edge metrics.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by dclusin on Saturday June 28, 2014 @01:27AM (#7957519)
But how will these leading-edge metrics enable us to deliver paradigm shifting solutions to our customers while simultaneously reducing costs and increasing operational efficiency?
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by source99 on Saturday June 28, 2014 @01:50AM (#7957552)
While making the world a better place!
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by msane on Saturday June 28, 2014 @02:16AM (#7957591)
That's the best part - it's automated with a context-sensitive customizability infrastructure. That means you can expand any targeted benchmark in a demand-driven mesh.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by jey on Saturday June 28, 2014 @12:55AM (#7957457)
Is this basically like Apache Spark in its programming model?
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by espeed on Saturday June 28, 2014 @01:20AM (#7957504)
Yes, it's like Spark (http://spark.apache.org/) and SparkStreaming (http://spark.apache.org/streaming/) combined.

Here are the relevant papers...

* FlumeJava (iterative, data-parallel pipelines like Spark): http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/F...

* MillWheel (fault-tolerant stream processing like SparkStreaming): http://research.google.com/pubs/pub41378.html

Pointers to the IO blog posts...

* "Reimagining developer productivity and data analytics in the cloud" http://googlecloudplatform.blogspot.com/2014/06/reimagining-...

* "Sneak peek: Google Cloud Dataflow, a Cloud-native data processing service" http://googlecloudplatform.blogspot.com/2014/06/sneak-peek-g...

The Dataflow-specific talks at Google IO 2014...

* Big data, the Cloud way: Accelerated and simplified https://www.youtube.com/watch?v=Y0Z58YQSXv0

* The dawn of "Fast Data" https://www.youtube.com/watch?v=TnLiEWglqHk

* Predicting the future with the Google Cloud Platform https://www.youtube.com/watch?v=YyvvxFeADh8

* Keynote (starts at Urs Hölzle's segment on Google Cloud) https://www.youtube.com/watch?v=wtLJPvx7-ys#t=6932

Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by jey on Saturday June 28, 2014 @01:26AM (#7957518)
Cool. Does this mean Google is moving away toward languages that allow for easier use and serialization of closures than in C++ and Java? (For example, Spark uses Scala natively.)
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by espeed on Saturday June 28, 2014 @02:08AM (#7957582)
Dataflow is language agnostic. The Java API is being released first, and more languages will follow.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by entrusted on Saturday June 28, 2014 @01:05AM (#7957477)
The main takeaway from this article is that the author, Yevgeniy Sverdlik, has demonstrably never worked with distributed computing systems.

The rest is buzzwords propping up sweeping ridiculous conclusions.

Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by miralabs on Saturday June 28, 2014 @12:40AM (#7957437)
"said it got too cumbersome once the size of the data reached a few petabytes."

I dont think there's a lot of companies where data would reach this huge. Anyone has any idea on how large a typical warehousing database is?

Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by jamesaguilar on Saturday June 28, 2014 @12:55AM (#7957458)
What do you mean, warehousing? Like, item tracking inside an actual warehouse? Hard to imagine spending more than a kB per unique item -- more per SKU, but less per individual object -- so even if you have 1M items being tracked, the total size would only be a gigabyte. Even if you had a billion unique things in your store, the resulting database would still fit on a single flash drive.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by seanp2k2 on Saturday June 28, 2014 @01:10AM (#7957488)
Given the context, it seems like "warehouse" in parent's comment was more specifically "data warehouse": http://en.m.wikipedia.org/wiki/Data_warehouse

While I couldn't quickly find anything that speaks to any kind of average or normal size of a data warehouse, this article mentions Facebook's being around 300PB: https://code.facebook.com/posts/229861827208629/scaling-the-...

Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by frozenport on Saturday June 28, 2014 @01:02AM (#7957471)
Its about what they need and not what other companies need.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by neckbeard on Saturday June 28, 2014 @01:35AM (#7957532)
For those that missed it, similar discussion on this a couple of days ago: https://news.ycombinator.com/item?id=7947782
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by dang on Saturday June 28, 2014 @02:15AM (#7957590)
Yes, it's the same story, so we'll call this thread a dupe.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by t1m on Saturday June 28, 2014 @01:01AM (#7957470)
Is it just me or is using 'cloud' in their product names just shark jumping.

Seriously.

Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by capkutay on Saturday June 28, 2014 @02:21AM (#7957604)
This sounds more like Google app engine's shot at amazon kinesis than anything else.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by CHY872 on Saturday June 28, 2014 @12:17AM (#7957392)
Sounds a bit like Dryad?
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by jpgvm on Saturday June 28, 2014 @01:18AM (#7957499)
Yay, marketing, yay.
Re:Google Dumps MapReduce in Favor of New Hyper-Scale Analyt... by infocollector on Saturday June 28, 2014 @12:49AM (#7957448)
More snake oil? :-)