Airflow and the Future of Data Engineering: A Q&A

Re:Airflow and the Future of Data Engineering: A Q&A by ktamura on Wednesday March 01, 2017 @04:20AM (#13762433)

If simplicity and non-Python-centricity matter, I encourage folks to look into Digdag [1][2].

It's Ansible for Workflow Management.

While both Luigi and Airflow (somewhat rightfully) assume the user to know/have affinity for Python, Digdag focuses on ease of use and helping enterprises move data around many systems.

If we learned one thing from today's S3 outage, it's not enough to use multiple cloud infrastructure providers: you should probably have your data in multiple cloud providers as well.

[1] https://www.digdag.io

[2] https://github.com/treasure-data/digdag

[ Parent | on HN ]

Re:Airflow and the Future of Data Engineering: A Q&A by caravel on Wednesday March 01, 2017 @02:46PM (#13767537)

[auhtor] Oh cool, I didn't know you guys released your solution yet, I remember demoing Airflow to you guys early on. Looks like it turned out great, congrats on the release!

[ Parent | on HN ]

Re:Airflow and the Future of Data Engineering: A Q&A by ktamura on Monday March 13, 2017 @06:21PM (#13863048)

Thanks! Airflow was/has been a great source of inspiration =)

[ Parent | on HN ]

Re:Airflow and the Future of Data Engineering: A Q&A by ubolonton_ on Wednesday March 01, 2017 @05:28AM (#13762719)

Thanks! I skimmed through the doc. It looks very clean from a high-level view.

[ Parent | on HN ]