Re:Airflow and the Future of Data Engineering: A Q&A
by caravel on Wednesday March 01, 2017 @11:15AM
(#13764905)
[author of Airflow here] as I wrote in another comment, I'd argue for a programmatic approach to workflows/dataflows as opposed to drag and drop. It turns out that code is a better abstraction for software:
https://medium.freecodecamp.com/the-rise-of-the-data-enginee...
I'd also argue for open source over proprietary, mostly to allow for a framework that is "hackable" and extensible by nature. You can also count on the community to build a lot of the operators & hooks you'll need (Airflow terms).
Re:Airflow and the Future of Data Engineering: A Q&A
by larsf on Wednesday March 01, 2017 @03:07PM
(#13767748)
You can use Composable's Fluent API to author and run dataflows in code. No GUI required. Composable's platform is open and completely extensible in sense that anyone can add applications or first class modules to the system, including swapping out some of the more internal components.
In Composable's DAG execution engine, you can pull in data from various sources (SQL, NoSQL, csv, json, restful endpoints, etc.) into our common data format. You can then easily transform, orchestrate, or analyze your data using our built-in Modules (blocks) or you can easily write your own. You can then view your resulting data all within the webapp.
Reading the comments, it seems like Composable supports a lot of the things people are asking for here that Airflow is lacking. Maybe check us out and let us know what you think!
For more information: Composable Site - https://composableanalytics.com/ Try it yourself - https://cloud.composableanalytics.com/ Composable's Blog - http://blog.composable.ai/