Because people use it. It's been around a while and has an ecosystem around it. (This is the interesting thing about AWS, they keep the focus on what people do use, rather than some opinionated idea of what they should use.)

fyi airflow 2.0 ships in the next few weeks, we're made a lot of improvements https://www.youtube.com/playlist?list=PLCi-q9vYo4x-PESoBcXN0...

It'd be really helpful if there were an article that folks could read.

Consuming video instead of text assumes that (1) I can hear and (2) I have the time to sit and listen at your speaking pace and (3) I'm in a place where I can sit with the audio turned up (4) I really want to expend the extra bandwidth for your video.

This trend towards posting video content instead of, rather than along-side, written content is pretty anti-accessbility.

Here's an article from the same source: https://www.astronomer.io/blog/introducing-airflow-2-0/

Do you have an article or something I can skim/scan more quickly? Not going to invest over an hour to watch those videos.

I am hoping Argo Workflows (https://argoproj.github.io/projects/argo) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.

I like Argo and have used both solutions, but I'm not yet convinced that YAML based workflows are superior to Airflow's Python Code as workflows.

Agree that YAML is not really ideal for this case.

For other readers Argo does support 'Scripts' which means you can implement complex logic similar to Airflow if required but can be a bit clumsy: https://argoproj.github.io/argo/examples/#scripts-results

Additionally, I think we may see other layers on top of Argo in future that may abstract from the YAML similar to how Kubeflow uses Argo for the Kubeflow Pipelines capability: https://www.kubeflow.org/docs/components/pipelines/pipelines...

Have you tried Flyte - flyte.org?

Running something on Kubernetes does not make this something highly available. It makes this something randomly restarting.

Maybe.

In this case Argo uses the Kubernetes distributed state store (etcd or equivalent) and is a stateless service so can deal with failures quite well.

Agreed. We've had good luck running all sorts of jobs on Argo and wrote up some of our experiences at https://www.interline.io/blog/scaling-openstreetmap-data-wor...

Agreed. It's the golden standard for data workflows right now, but it's pretty cumbersome to work with.

I'm building a SaaS product to try and fix the issues I found with Airflow et al. Feel free to check my profile/reach out if you're interested in trying an alternative.

I have read Airflow 2.0 release addresses lot of long-term cirticisms of Airflow. Hope anyone here experience with Airflow can illuminate us.

Me too. It's so complicated! If you just need to schedule something without any dependencies, maybe try: https://github.com/maxhumber/hickory

Disclaimer: I'm the creator.

Lmao this is a great comment - I have found so many bizarre eccentricities with Airflow, but it is still, in general, quite nice to build with. What are you thinking about?

Curious what you like better?

Luigi from Spotify:

https://github.com/spotify/luigi

We’ve been using it for complex update workflows for about 5 yrs now, and it just works.

It doesn’t do scheduling or have a fancy ui, but it’s a solid workhorse.

Spotify is moving to https://Flyte.org and building a Luigi to Flyte compiler. Stay tuned

Although this article [1] compares Airflow to Prefect it outlines pretty interestingly limitations of Airflow.

I would be also curious to hear more about alternative and mature open-source solutions to Airflow.

[1] https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...

I would recommend you check out Airflow 2.0. It's a pretty major rebuild in a whole lot of ways (new UI, new DAG API, up to TEN TIMES faster task execution, multiple schedulers at once). I've actually had friends prepared to pick Prefect over Airflow until they tried 2.0. We put a lot of work into it, including extensive QA time to ensure that it runs reliably.

Disclosure: I'm on the Airflow PMC.

Is there a list of backward incompatible changes?

Also, how would you describe the overall health of the code base with all of these new features added?

Thanks for your work on airflow. 2.0 is exciting and we are looking forward to it. Love airflow 1 though too, it has helped us a ton.

not OP, but I prefer https://github.com/azkaban/azkaban to airflow

so, workflows described in yaml? I also am a bit ambivalent about Airflow and the way workflows are built up in it, but YAML seems really weak and unlikely to handle complexity well.

Also - it's entirely built around Hadoop from what I can see? Seems a limited use case compared to Airflow.

You raise a good point, there is an apparent need for an application that retains the dags-as-code but radically simplifies the airflow architecture.

if you like YAML-based DAGs, you can do that in Airflow with the dag-factory extension https://github.com/ajbosco/dag-factory

temporal.io

This is it, the DAG model is a thing of the past