Joining a billion rows 20x faster than Apache Spark

Re:Joining a billion rows 20x faster than Apache Spark by placeybordeaux on Thursday February 09, 2017 @05:17PM (#13611543)

The picture has now gotten a little fuzzier as this blog post conflates map reduce and YARN and calls them both hadoop. The scala pseudo code is just about exactly what you'd use with spark which runs on YARN.

[ Parent | on HN ]

Re:Joining a billion rows 20x faster than Apache Spark by edw on Thursday February 09, 2017 @05:23PM (#13611589)

I think his point is that bloated, over-engineered Big Data systems—whether batch or streaming—are overkill for the vast majority of problems.

[ Parent | on HN ]

Re:Joining a billion rows 20x faster than Apache Spark by placeybordeaux on Thursday February 09, 2017 @05:50PM (#13611775)

There are just many points that don't really apply to stuff like spark or tez that runs on YARN:

ex: Hadoop << SQL, Python Scripts

I completely agree with

Mapreduce << SQL, Python Scripts

I do a lot of my processing on sparkSQL and through RDD transformations as opposed to Mapreduce limiting, slow KV style processing.

[ Parent | on HN ]