Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly
by ibishvintilli on Saturday October 09, 2021 @08:36PM
(#28814703)
This is a very bad idea. You cannot hide ETL in a Rest API, at most you can hide the T part and it's a lot more work. you need to build an extra service that reads the data from a database and does some padgination. It's way more complex than a one-liner to get data from a jdbc connection and way much slower.
Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly
by kdazzle on Friday October 08, 2021 @08:16PM
(#28805729)
I used to do that, but having to deal with timeouts in long running jobs was frustrating because i never knew if something was successful. Could bump up the timeouts, but theyre there for a reason.
2 - Use PythonOperator to call your APIs.
It's working really well for us.