News for hackers. Stuff that matters.
Re:Using an ETL framework vs. writing yet another ETL script by Cwizard on Friday December 17, 2021 @02:17PM (#29597020)
Do you use some kind of library to help with transformations? If you have some volume using plain python can become a bottleneck no? Any tips in this area would be appreciated.
Re:Using an ETL framework vs. writing yet another ETL script by WhyCause on Friday December 17, 2021 @02:36PM (#29597300)
I'm not the person you were asking, but I, too, use mostly Python for ETL tasks as a good chunk of my job.

Python's speed, or lack thereof, is rarely an issue. Most tasks that I write/run finish in less than a minute, including the time for transformations. Some of those can get fairly hairy, as well. Most of the time is spent querying the DB and uploading to the remote.

For the tasks where the transformations are too complex, you can greatly decrease the time necessary by tweaking the queries (i.e., limiting rows to just changed rows, etc.). And frankly, once you realize most data doesn't need to update in real time, it doesn't matter how long the transformation step takes (as long as it's fewer than about 23 1/2 hours).

Re:Using an ETL framework vs. writing yet another ETL script by davinchia on Sunday December 19, 2021 @03:07AM (#29612280)
Airbyte engineer here. We use Dbt to normalise the delivered data at the final destination warehouse.

If it's not a destination warehouse, how transformations are handled depends on the connector implementation.

Re:Using an ETL framework vs. writing yet another ETL script by elchief on Friday December 17, 2021 @02:39PM (#29597341)
pyspark if big-ol' data sets

pandas otherwise