Here is, a slightly outdated, article that compares several ETL workflow tools http://bytepawn.com/luigi-airflow-pinball.html . Why we choose Airflow was because of the following reasons:

* Scheduler that knows how to handle retries, skipped tasks, failing tasks

* Great UI

* Horizontal scaleable

* Great community

* Extensible; we could make it work in an enterprise context (kerberos, ldap etc)

* No XML

* Testable and debug-able workflows