Re:Airflow and the Future of Data Engineering: A Q&A
by tedmiston on Wednesday March 01, 2017 @11:25AM
(#13765051)
I definitely get where you're coming from. At Astronomer, we use both Airflow and Spark, though Spark is very new to me.
For us, Airflow manages workflows and task dependencies but all of the actual work is done externally. Each task (operator) runs whatever dockerized command with I/O over XCom. Note that we use a custom Mesos executor instead of the Celery executor. An Airflow DAG might kick off a different Spark job based on upstream tasks.
For us, Airflow manages workflows and task dependencies but all of the actual work is done externally. Each task (operator) runs whatever dockerized command with I/O over XCom. Note that we use a custom Mesos executor instead of the Celery executor. An Airflow DAG might kick off a different Spark job based on upstream tasks.