News for hackers. Stuff that matters.
Re:Data Engineering Patterns with Apache Airflow [video] by rhombocombus on Wednesday August 29, 2018 @01:19PM (#17869964)
How does it compare to big iron enterprise ETL tools like IBM datastage? I have only dabbled but it looks far more appealing for a variety of reasons.
Re:Data Engineering Patterns with Apache Airflow [video] by caravel on Wednesday August 29, 2018 @03:27PM (#17871005)
For context, I used Datastage, Informatica, Ab Initio and SSIS in previous lives an went on to write the first version of Airflow. I developed a taste for pipelines-as-code while working at Facebook using an internal tool that is not open source.

I'd argue that pipelines as code, as opposed to dragndrop GUIs, is a better approach, at least for people who are comfortable writing code. Code is easy to version, test, diff, collaborate and allows for the creation of arbitrary abstractions.

Re:Data Engineering Patterns with Apache Airflow [video] by philosopherlawr on Wednesday August 29, 2018 @02:40PM (#17870558)
ETL tools just can't compete with a tool that forces code to do anything. It might seem backwards, but we've abandoned all non-code environments and force pure-code for configuration for all of our pipelines.