Re:Dabbling with Dagster vs. Airflow
by KptMarchewa on Wednesday September 14, 2022 @05:49PM
(#32843886)
>On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players.
Well, can't disagree with that.
>There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.
Any link to that? I can't find the story.
On one hand, it is immensely successful Apache project.
On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players.
There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.