Re:Open Source Python ETL
by thibautdr on Tuesday June 18, 2024 @11:32PM
(#40724493)
Thanks for your comment! I do believe it depends on who you ask and ultimately both will co-exist. I also think low-code solutions democratize access to ETL development offering a significant productivity advantage for smaller teams. With Amphi, I'm trying to avoid the common pitfalls of other low-code ETL tools, such as scalability issues, inflexibility, and vendor lock-in, while embracing the advantages of modern ETL-as-code:
- Pipelines are defined as JSON files (git workflow available)
- Generates non-proprietary Python code: This means the pipelines can be deployed anywhere, such as AWS Lambda, EC2, on-premises, or Databricks.
[ Parent
| on HN ]
Re:Open Source Python ETL
by banku_brougham on Wednesday June 19, 2024 @01:31AM
(#40725142)
Im very leery of low code, but I like the idea of ETL defined as configuration.
Re:Open Source Python ETL
by stoperaticless on Wednesday June 19, 2024 @01:56AM
(#40725267)
Etl as text is good, because you can save it in version control. (Is it “code” or “json” is irrelevant for the vcs)
Edit: save in vcs stringly implies usability of ‘diff’ and ‘grep’