Re:Introduction to Apache Airflow
by contravariant on Friday May 29, 2020 @11:28AM
(#23351105)
Sensors are quite helpful when you don't know the exact moment your data will come in.
[ Parent
| on HN ]
I think it's better to place everything in one DAG if that solves the problem. If it doesn't then sensors are ok I guess, but I would try to avoid them otherwise.
I really disagree here. Monolithic dags are a bigger pain to manage.
Breaking them out into smaller dags makes retrying/backfilling/etc a lot more straightforward. It also lets you reuse those pieces more easily.
We have compute DAGs that are the upstream dependency for many other dags. Originally, this dag was monolithic and loaded data into one table. But because the dag is split into computation and loading we can easily add more downstream dags without changing how the first one operated.