Re:Launch HN: DAGWorks – ML platform for data science teams
by rubenfiszel on Tuesday March 07, 2023 @03:30PM
(#35060765)
Ola, I'm the founder of windmill [1] which is an OSS orchestration platform for scripts, including python, and I'm pretty sure one could compile an hamilton dag to a windmill openflow very simply (we use a mounted folder in `./shared` to share heavy data across steps). Right now, one can do ETL using polars on windmill but i'd love to have an even more structure code framework to recommend.
We should chat!
Re:Launch HN: DAGWorks – ML platform for data science teams
by krawczstef on Tuesday March 07, 2023 @03:47PM
(#35060986)
Nice! Yep that's part of our vision to "compile onto", i.e. generate code for, frameworks such as yours.
Happy to sketch something out - want to join our slack to chat https://join.slack.com/t/hamilton-opensource/shared_invite/z...?
1. Integrate with orchestration systems (E.G. run Hamilton pipelines on different orchestration platforms). You could imagine compiling to airflow pipelines, running hamilton on metaflow, compiling to a vertex pipeline etc...
2. Adapters to load data from/save data to external providers. E.G. logging data quality to whylogs, loading data up from snowflake, saving a model to mlflow...
We're actively working on building this out though -- have some partners who hare helping build out use-cases!