Prefect has more polish and is easier to get started than any of the existing options. We've been running their self-hosted for over three years and it basically stays out of the way.

We looked at Dagster as well as Airflow. I really, really liked Dagster but the BI team didn't.

I cannot imagine using Airflow for anything meaningful and respecting myself at the end of a work day. The local development experience was abysmal. Deployments sucked.

That being said, if you're not using anything except maybe cron right now, and if you don't care about the solution being a proper data pipelines orchestration platform trademark symbol, I'd recommend starting with Windmill.

I've done a lot evaluation of such frameworks, and I hope to publish more on it. It really depends on your requirements. I would look at Prefect, Flyte, Dagster, and Temporal ahead of Airflow, though.

Airflow is kind of the default platform. If you do not have a goto alternative which is superior in multiple dimensions today, I think that says the problem itself is hard.

Might as well go with the devil everyone knows.

Where do you think you'd publish it? I'm interested to read the details.

We've been running Dagster for ~6 months on ECS and it has been rock solid. (knocks on wood)

jenkins