Scrap your MapReduce – Introduction to Apache Spark

Re:Scrap your MapReduce – Introduction to Apache Spark by dekhn on Monday November 17, 2014 @10:41AM (#8618649)

MSFT killed Naiad's predecessor Dryad in favor of Hadoop some time ago, because Hadoop was becoming popular. The primary author linked in the page works at "Microsoft Silicon Valley"which was just shut down and in fact now lists himself on LinkedIn as "Researcher At Large, previously at Microsoft"

So, how do we know Naiad has much future? . Technologically, it may be better/more reliable/faster, but if it's a niche product that gets desupported just because it never took off... it doesn't really matter.

Spark on the other hand has a great deal of momentum and in my experience, momentum and adoption trump technical elegance in the short run...

(don't get me wrong: I thought Dryad was awesome. Google's Flume is very similar in some ways. MapReduce's days are numbered except for a small number of problems which can't be easily ported).

[ Parent | on HN ]

Re:Scrap your MapReduce – Introduction to Apache Spark by deadgrey19 on Monday November 17, 2014 @11:14AM (#8618804)

All good points. I can't say what the future of Naiad is. What they have done to Microsoft Research Silicon Valley is disgusting (I worked there too for a short time).

In our experiences the performance claims with Spark have been more hype than substance. Naiad on the other hand has been hard to find a corner case for.

Naiad is open source licensed under an Apache License so one can only hope...

[ Parent | on HN ]

Re:Scrap your MapReduce – Introduction to Apache Spark by deadgrey19 on Tuesday November 18, 2014 @06:33AM (#8623451)

FYI: Link to Naiad github repo: https://github.com/MicrosoftResearch/Naiad

[ Parent | on HN ]

Re:Scrap your MapReduce – Introduction to Apache Spark by dekhn on Tuesday November 18, 2014 @11:43AM (#8624738)

Thanks, but: this is a dead project.

Also, it appears to be tied to Windows (it's delivered as a VS solution).

[ Parent | on HN ]