A "quickly-implemented" feature != parity in utility. E.g. Airflow's TaskFlow superficially looks like some of Dagster's APIs, but the experience of using them is way different:
- TaskFlow is built on top of XCom, which isn't designed for data sizes that larger than small.
- If you use TaskFlow instead of the recommended practice of using Kubernetes Operators (https://medium.com/bluecore-engineering/were-all-using-airfl...), you lose the ability to isolate Python dependencies between DAGs.
Having worked on Hadoop in the past, I saw how it's easy for large Apache projects to add tons of new features to get "parity" with upstart competitors, but still lose to software like Snowflake and Spark that engineer those features in the "right" way as part of a streamlined product vision.
On one hand, it is immensely successful Apache project.
On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players.
There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.
Well, can't disagree with that.
>There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.
Any link to that? I can't find the story.
I can't find it either, I'm pretty sure I heard it on some conference, so consider it made up. It was years ago anyway.
Much of what makes Spark successful was there at the beginning - e.g. a clean programming model and an architecture that didn't require provisioning new resources from the resource manager every time it launches a task.
1. Dagster and Prefect won't ever have a hosted version on AWS/GCP (teams can spin these up from their consoles without even considering a vendor conversation or leaving their infra).
2. There are very large Airflow projects out there with huge DE teams running them, that just can't really move to Dagster or Prefect very easily. Not do they really want to if they can do what they need to in a tool they are familiar with.
My perspectives have been very much coming at the tools with fresh eyes.
Will they overtake Airflow in terms of features? They probably already have.
Also, as other's have mentioned the comparison to MWAA is unfair, the true own of Airflow is Astronomer as they have over 50% of the commits to Airflow, and Astronomer is a much better product the MWAA.