Airflow and the Future of Data Engineering: A Q&A

Re:Airflow and the Future of Data Engineering: A Q&A by botswana99 on Wednesday March 01, 2017 @11:36AM (#13765201)

[Bias Alert: I'm Head Chef of DataKitchen]. Our perspective is that the DAG abstraction should not apply only to data engineering, but the whole analytic process of data engineering, data science, and data visualization. Analytic teams love to work with their favorite tools -- Python, SQL, ETL, Jupyter, R, Tableau, Alteryx, etc. The question is how do you get those diverse teams and tools to work together to deliver fast, with high quality, and reusable components?

We've identified seven steps taken from DevOps, CI, Agile and Lean Manufacturing (https://www.datakitchen.io/platform.html#sevensteps) that you can start to apply today. We also created a 'DataOps' platform that incorporates those principles into a software: https://www.datakitchen.io.

The challenge is that there are many separate DAGs (and code and configuration) involved in producing complete production analytics embedded in each of the tools the team has selected. So what is needed is a “DAG of DAGs” that encompasses the whole analytic tool chain.

[ Parent | on HN ]

Re:Airflow and the Future of Data Engineering: A Q&A by caravel on Wednesday March 01, 2017 @12:33PM (#13765996)

[Bias Alert: author of Airflow] can confirm that Airflow allows you to incorporate all of the seven steps, and more as an open platform.

At Airbnb Airflow is far from being limited to data engineering. All the batch scheduling goes through Airflow and many team (data science, analysts, data infra, ML infra, engineering as a whole, ...) uses Airflow in all sorts of ways.

Airflow has a solid story in terms of reusable components, from extendable abstractions (operator, hooks, executors, macros, ...) all the way to computation frameworks.

[ Parent | on HN ]