News for hackers. Stuff that matters.
Re:Kedro – Creating reproducible, maintainable and modular d... by waylonwalker on Saturday January 22, 2022 @12:56PM (#30039049)
I use it quite heavily at my day job. The core pieces of kedro are indispensable on large projects (Pipeline, Catalog, DataSets, Runner, hooks). The marketing has a lot of focus on the template, It's a great template, but not the key feature that brings me back to it. Really like that I never have to worry about io, the catalog abstracts this away. I don't worry about long runs to get started with my work. It saves all intermediate steps automatically, and the Pipeline (DAG) lets me run just the parts that I need for my task. Before using kedro I would need to scaffold out a way to save all of these intermediate steps, or suffer from long run times to get started on my task. Lastly the DAG is indispensible during environment migrations, it can quickly tell me all of its edges that I need to be concerned with picking up and moving with me.

Unlike similar projects kedro is just a python framework. It lets me build, deploy, and orchestrate however works best for my team.

Re:Kedro – Creating reproducible, maintainable and modular d... by barefeg on Saturday January 22, 2022 @03:47PM (#30040676)
Could you point me to the component responsible for storing the intermediate data of the DAG run? I was looking for this but couldn’t find it from quickly scanning the docs
Re:Kedro – Creating reproducible, maintainable and modular d... by joelschw on Saturday January 22, 2022 @04:04PM (#30040816)
Re:Kedro – Creating reproducible, maintainable and modular d... by waylonwalker on Sunday January 23, 2022 @09:54AM (#30046534)
By adding a catalog entry for each of your datasets with a filepath argument, it will save them to that place. All their datasets use fsspec under the hood so this path can be an s3 bucket or a local path and it all just works.
Re:Kedro – Creating reproducible, maintainable and modular d... by kjkjadksj on Sunday January 23, 2022 @12:01AM (#30043773)
Is this different from snakemake?
Re:Kedro – Creating reproducible, maintainable and modular d... by joelschw on Sunday January 23, 2022 @12:58AM (#30043967)
Some similar concepts, but Kedro is specialised on the ML engineering workflow and team collaboration