News for hackers. Stuff that matters.
Re:Open Source Python ETL by gregw2 on Wednesday June 19, 2024 @09:07AM (#40727934)
Isn’t pandas centric ETL much more memory intensive and less compute efficient than using SQL?
Re:Open Source Python ETL by rkozik1989 on Wednesday June 19, 2024 @09:26AM (#40728091)
That's kind of the tradeoff you make with any low-code/no-code technology. You leverage prebuilt components and string them together to achieve some kind of task. Which isn't most efficient thing in the world to do but it does work assuming you have enough compute resources to throw at it, and return what you generally achieve is an end product that's completed faster than the traditional development route.

You could just use SQL but then you'd have to develop and test the entire infrastructure to support your component-oriented architecture from scratch, and at that point you're kind of just reinventing the wheel because that's basically just pandas with less features.

Low-code is kind of just Authorware for a new generation... assuming you're old enough to remember that technology.

Re:Open Source Python ETL by thibautdr on Monday June 24, 2024 @05:26PM (#40781001)
I wrote an article questioning the use of Pandas for ETL. I invite you to read it: https://medium.com/@thibaut_gourdel/should-you-use-pandas-fo...