News for hackers. Stuff that matters.
Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly by pid-1 on Friday October 08, 2021 @01:50PM (#28802021)
1 - Hide your ETL logic in REST APIs and use common microservice patterns for monitoring, logging, etc...

2 - Use PythonOperator to call your APIs.

It's working really well for us.

Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly by ibishvintilli on Saturday October 09, 2021 @08:36PM (#28814703)
This is a very bad idea. You cannot hide ETL in a Rest API, at most you can hide the T part and it's a lot more work. you need to build an extra service that reads the data from a database and does some padgination. It's way more complex than a one-liner to get data from a jdbc connection and way much slower.
Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly by kdazzle on Friday October 08, 2021 @08:16PM (#28805729)
I used to do that, but having to deal with timeouts in long running jobs was frustrating because i never knew if something was successful. Could bump up the timeouts, but theyre there for a reason.