On one hand, it is immensely successful Apache project.
On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players.
There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.
Well, can't disagree with that.
>There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.
Any link to that? I can't find the story.
I can't find it either, I'm pretty sure I heard it on some conference, so consider it made up. It was years ago anyway.
Much of what makes Spark successful was there at the beginning - e.g. a clean programming model and an architecture that didn't require provisioning new resources from the resource manager every time it launches a task.
A "quickly-implemented" feature != parity in utility. E.g. Airflow's TaskFlow superficially looks like some of Dagster's APIs, but the experience of using them is way different:
- TaskFlow is built on top of XCom, which isn't designed for data sizes that larger than small.
- If you use TaskFlow instead of the recommended practice of using Kubernetes Operators (https://medium.com/bluecore-engineering/were-all-using-airfl...), you lose the ability to isolate Python dependencies between DAGs.
Having worked on Hadoop in the past, I saw how it's easy for large Apache projects to add tons of new features to get "parity" with upstart competitors, but still lose to software like Snowflake and Spark that engineer those features in the "right" way as part of a streamlined product vision.