Yeah, I don't mean to knock large Apache projects in general - they can be great software. My point is that it's hard for them to change / successfully copy their competitors.

Much of what makes Spark successful was there at the beginning - e.g. a clean programming model and an architecture that didn't require provisioning new resources from the resource manager every time it launches a task.