News for hackers. Stuff that matters.
Re:Open Source Python ETL by thibautdr on Tuesday June 18, 2024 @08:43PM (#40723576)
Hi everyone, thanks for posting Amphi :)

To give some context, Amphi is a low-code ETL tool for both structured and unstructured data. The key use cases include file integration, data preparation, data migration, and creating data pipelines for AI tasks like data extraction and RAG. What sets it apart from traditional ETL tools is that it generates Python code that you own and can deploy anywhere. Amphi is available as a standalone web app or as a JupyterLab extension.

Visit the GitHub: https://github.com/amphi-ai/amphi-etl Give it a try and let me know what you think

Re:Open Source Python ETL by slt2021 on Wednesday June 19, 2024 @12:01AM (#40724665)
i liked the idea of leveraging jupyterlab as server. data engineers/scientists already use jupyter, so this is neat idea.

custom extension for jupyterlab is a great way to leverage existing jupyterlab install base: not everyone will be willing to install and jump through hoops to install software X, but installing extension is one pip install away and no need to run separate process, since you are running inside jupyterlab server.

this reminds of ALTERYX (another drag and drop ETL tool)

Re:Open Source Python ETL by thibautdr on Wednesday June 19, 2024 @12:11AM (#40724727)
Thanks! Being based on JupyterLab also allows Amphi to benefit from the vast ecosystem of extensions already available, such as the Git extension or using different file systems (S3).

Some users pointed out they were Alteryx users but liked the Python code generation from Amphi :)

Re:Open Source Python ETL by slt2021 on Wednesday June 19, 2024 @12:19AM (#40724767)
just an idea: is it possible to code generate Airflow code? since a lot of companies use airflow as ETL orchestrator
Re:Open Source Python ETL by thibautdr on Wednesday June 19, 2024 @02:46AM (#40725527)
Amphi generates Python code, so you can definitely orchestrate them through Airflow but it doesn't generate "Airflow code" so to speak. Now, in the future we might develop Airflow specific workflows or maybe operators.
Re:Open Source Python ETL by johhns4 on Wednesday June 19, 2024 @06:22AM (#40726819)
Wow amazing work! How does the inputs work, are they created for you or does it support custom as well?
Re:Open Source Python ETL by thibautdr on Wednesday June 19, 2024 @02:02PM (#40730779)
Thank you! Inputs components are pre-built for now but the ability to add custom inputs is coming soon!
Re:Open Source Python ETL by johhns4 on Monday June 24, 2024 @04:44AM (#40773744)
Do please say when this will work as this could make my workflows a lot easier to visualize and work with.
Re:Open Source Python ETL by thibautdr on Monday June 24, 2024 @05:28PM (#40781020)
I sure will!
Re:Open Source Python ETL by OutOfHere on Wednesday June 19, 2024 @03:57PM (#40731835)
You know what does not set it apart? AI-washing. Also, lying about being open source when it isn't.
Re:Open Source Python ETL by isjamesalive on Thursday June 20, 2024 @09:59AM (#40738878)
To be fair, the only place the words ‘open’ and ‘source‘ appear in the readme are once in a sub-heading, where it’s phrased ‘open-source’. It’s clearly labelled ELv2.

Possibly more of a subtle miscommunication or misunderstanding than a deliberate lie.

Re:Open Source Python ETL by OutOfHere on Thursday June 20, 2024 @10:57AM (#40739459)
Don't kid yourself. The title of this submission itself starts with "Open Source". Moreover, the author has made the explicit decision to not fix the readme.