Open Source Python ETL

Re:Open Source Python ETL by vekker on Wednesday June 19, 2024 @02:33AM (#40725463)

Does this also manage the infrastructure side of ETL? Usually some parts in a complex ETL process take a lot more processing power, so are run on different machines. From a quick glance at this, it seems like a WYSIWYG ETL tool for running ETL jobs on one machine?

[ Parent | on HN ]

Re:Open Source Python ETL by thibautdr on Wednesday June 19, 2024 @02:53AM (#40725563)

Thanks for your question. Amphi generates Python code using Pandas and can scale on a single machine or even multiple machines using Modin, but the process is manual for now. Future plans include deploying pipelines on Spark clusters and other services such as Snowflake.

[ Parent | on HN ]

Re:Open Source Python ETL by Kalanos on Wednesday June 19, 2024 @05:24PM (#40732532)

what about dask?

[ Parent | on HN ]

Re:Open Source Python ETL by thibautdr on Wednesday June 19, 2024 @05:31PM (#40732594)

Using Modin, deploying the pandas code on Dask should be possible: https://modin.readthedocs.io/en/stable/development/using_pan...

[ Parent | on HN ]