rs_atl

You have a point, but it's also a fact that lots of people use "Hadoop" interchangeably with "MapReduce". And Spark can in fact replace the Hadoop infrastructure entirely, as it's not a component of that ecosystem. Just because the various Hadoop vendors also support Spark only validates the point that there's a need to "go beyond Hadoop".

Long winded in more ways than one. I would still use Spark even if it were slower than Hadoop, just to get a sane API.

It can also spill to disk if needed, which is what Hadoop always does.