It is a bit more nuanced.

On one hand, it is immensely successful Apache project.

On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players.

There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.

>On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players.

Well, can't disagree with that.

>There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.

Any link to that? I can't find the story.

> Any link to that? I can't find the story.

I can't find it either, I'm pretty sure I heard it on some conference, so consider it made up. It was years ago anyway.

Yeah, I don't mean to knock large Apache projects in general - they can be great software. My point is that it's hard for them to change / successfully copy their competitors.

Much of what makes Spark successful was there at the beginning - e.g. a clean programming model and an architecture that didn't require provisioning new resources from the resource manager every time it launches a task.

probably meant databricks