Airflow's Problem

Re:Airflow's Problem by edumucelli on Tuesday August 02, 2022 @09:39AM (#32318321)

More recent tools such as Dagster and Prefect have much more to offer. One simple example is communication between tasks. Airflow has a clunky system for that called XCom. The actual author of XCom says you probably should not use it due to the level of hackery it has under the hood [1]:

On Dagster and Prefect you communicate between tasks as if you were writing pure Python. On Airflow on the other hand ...

[1] https://www.youtube.com/watch?v=TlawR_gi8-Y&t=740s

[ Parent | on HN ]

Re:Airflow's Problem by marcinzm on Tuesday August 02, 2022 @09:42AM (#32318346)

I'm not sure if linking to a talk from 2018 in 2022 for a project that is being actively worked on (a bunch of Python abstractions for xcom were added in 2.0) is fair.

[ Parent | on HN ]

Re:Airflow's Problem by byteflip on Tuesday August 02, 2022 @12:28PM (#32320820)

Yes, curious if the Taskflow API introduced as part of Airflow 2.0 reduces this pain. It appears much easier/saner than working with XCOMs directly - less coupling and removes the need for lots of boilerplate code.

[ Parent | on HN ]

Re:Airflow's Problem by karog on Tuesday August 02, 2022 @12:36PM (#32320933)

There's also Flyte, which is natively data aware and schedules tasks around data dependencies. The syntax is essentially pure python too.

[ Parent | on HN ]