News for hackers. Stuff that matters.
Re:Open Source Python ETL by thibautdr on Tuesday June 18, 2024 @11:32PM (#40724493)
Thanks for your comment! I do believe it depends on who you ask and ultimately both will co-exist. I also think low-code solutions democratize access to ETL development offering a significant productivity advantage for smaller teams. With Amphi, I'm trying to avoid the common pitfalls of other low-code ETL tools, such as scalability issues, inflexibility, and vendor lock-in, while embracing the advantages of modern ETL-as-code: - Pipelines are defined as JSON files (git workflow available) - Generates non-proprietary Python code: This means the pipelines can be deployed anywhere, such as AWS Lambda, EC2, on-premises, or Databricks.
Re:Open Source Python ETL by banku_brougham on Wednesday June 19, 2024 @01:31AM (#40725142)
Im very leery of low code, but I like the idea of ETL defined as configuration.
Re:Open Source Python ETL by stoperaticless on Wednesday June 19, 2024 @01:56AM (#40725267)
Etl as text is good, because you can save it in version control. (Is it “code” or “json” is irrelevant for the vcs)

Edit: save in vcs stringly implies usability of ‘diff’ and ‘grep’