Re:Airflow and the Future of Data Engineering: A Q&A
by smooc on Wednesday March 01, 2017 @11:03AM
(#13764772)
Airflow uses Celery to horizontally scale its execution. The Airflow scheduler takes care of what tasks to run in what order, but also what to do when they fail, need to retry, don't need to run at all, backfill the past etc.
Spark for Airflow is just one of the engines where a transformation of data can happen.
Spark for Airflow is just one of the engines where a transformation of data can happen.