Re:Open Source Python ETL
by thibautdr on Wednesday June 19, 2024 @02:53AM
(#40725563)
Thanks for your question. Amphi generates Python code using Pandas and can scale on a single machine or even multiple machines using Modin, but the process is manual for now. Future plans include deploying pipelines on Spark clusters and other services such as Snowflake.
[ Parent
| on HN ]
what about dask?
Using Modin, deploying the pandas code on Dask should be possible: https://modin.readthedocs.io/en/stable/development/using_pan...