News for hackers. Stuff that matters.
Re:Airflow and the Future of Data Engineering: A Q&A by caravel on Wednesday March 01, 2017 @02:44PM (#13767509)
At a high level [for Airflow specifically] the scheduler or workflow engine cares most about the tasks and their dependencies, and is somewhat agnostic about the units of work it triggers.

It's possible to use a feature called XCom as a message bus between tasks, but would typically direct people in the direction of having stateless, idempotent, "contained" units of work and avoid cross task communication as much as possible. https://airflow.incubator.apache.org/concepts.html#xcoms

For your case [which I have little input on] I think singleton DAGs described in another post on this page may work.

There are no comments on this comment yet.