News for hackers. Stuff that matters.

A comment on ETL Pipelines with Airflow: The Good, the Bad and the Ugly.

Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly by 41b696ef1113 on Friday October 08, 2021 @06:33PM (#28804900)
Has anyone kicked the tires on Airflow, Prefect, and Dagster and care to give their thoughts? My initial foray into Airflow 1 met a lot of complexity that both Prefect and Dagster claim to minimize.
[ Parent | on HN ]
Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly by FridgeSeal on Friday October 08, 2021 @10:03PM (#28806355)
Tried some of them, threw them away for various reasons and used Argo Workflows instead.

Airflow mixes orchestration and the actual transformation, and I wasn’t a fan of being ao intimately tied to Python. Argo let’s us run arbitrary containers (a mix of Spark, Rust, etc) and played nicer with the tools we were already using (Kafka, K8s etc), didn’t require us to write Python and cleanly separated orchestration and actual ETL/ELT work. It’s working fantastically for us.

[ Parent | on HN ]
Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly by mwadhera on Saturday October 09, 2021 @10:51AM (#28810383)
looking at Argo for exactly these reasons. open to chat? (didn’t see your email in profile)
[ Parent | on HN ]