News for hackers. Stuff that matters.
Re:Introduction to Apache Airflow by contravariant on Friday May 29, 2020 @11:28AM (#23351105)
Sensors are quite helpful when you don't know the exact moment your data will come in.
Re:Introduction to Apache Airflow by simo7 on Friday May 29, 2020 @11:39AM (#23351267)
I think it's better to place everything in one DAG if that solves the problem. If it doesn't then sensors are ok I guess, but I would try to avoid them otherwise.
Re:Introduction to Apache Airflow by prions on Friday May 29, 2020 @11:45AM (#23351359)
I really disagree here. Monolithic dags are a bigger pain to manage.

Breaking them out into smaller dags makes retrying/backfilling/etc a lot more straightforward. It also lets you reuse those pieces more easily.

We have compute DAGs that are the upstream dependency for many other dags. Originally, this dag was monolithic and loaded data into one table. But because the dag is split into computation and loading we can easily add more downstream dags without changing how the first one operated.