Luigi from Spotify:

https://github.com/spotify/luigi

We’ve been using it for complex update workflows for about 5 yrs now, and it just works.

It doesn’t do scheduling or have a fancy ui, but it’s a solid workhorse.

Spotify is moving to https://Flyte.org and building a Luigi to Flyte compiler. Stay tuned

Although this article [1] compares Airflow to Prefect it outlines pretty interestingly limitations of Airflow.

I would be also curious to hear more about alternative and mature open-source solutions to Airflow.

[1] https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...

I would recommend you check out Airflow 2.0. It's a pretty major rebuild in a whole lot of ways (new UI, new DAG API, up to TEN TIMES faster task execution, multiple schedulers at once). I've actually had friends prepared to pick Prefect over Airflow until they tried 2.0. We put a lot of work into it, including extensive QA time to ensure that it runs reliably.

Disclosure: I'm on the Airflow PMC.

Is there a list of backward incompatible changes?

Also, how would you describe the overall health of the code base with all of these new features added?

Thanks for your work on airflow. 2.0 is exciting and we are looking forward to it. Love airflow 1 though too, it has helped us a ton.

not OP, but I prefer https://github.com/azkaban/azkaban to airflow

so, workflows described in yaml? I also am a bit ambivalent about Airflow and the way workflows are built up in it, but YAML seems really weak and unlikely to handle complexity well.

Also - it's entirely built around Hadoop from what I can see? Seems a limited use case compared to Airflow.

You raise a good point, there is an apparent need for an application that retains the dags-as-code but radically simplifies the airflow architecture.

if you like YAML-based DAGs, you can do that in Airflow with the dag-factory extension https://github.com/ajbosco/dag-factory

temporal.io

This is it, the DAG model is a thing of the past