I use it quite heavily at my day job. The core pieces of kedro are indispensable on large projects (Pipeline, Catalog, DataSets, Runner, hooks). The marketing has a lot of focus on the template, It's a great template, but not the key feature that brings me back to it. Really like that I never have to worry about io, the catalog abstracts this away. I don't worry about long runs to get started with my work. It saves all intermediate steps automatically, and the Pipeline (DAG) lets me run just the parts that I need for my task. Before using kedro I would need to scaffold out a way to save all of these intermediate steps, or suffer from long run times to get started on my task. Lastly the DAG is indispensible during environment migrations, it can quickly tell me all of its edges that I need to be concerned with picking up and moving with me.

Unlike similar projects kedro is just a python framework. It lets me build, deploy, and orchestrate however works best for my team.

Could you point me to the component responsible for storing the intermediate data of the DAG run? I was looking for this but couldn’t find it from quickly scanning the docs

Here you go - https://kedro.readthedocs.io/en/0.17.6/09_development/03_com...

By adding a catalog entry for each of your datasets with a filepath argument, it will save them to that place. All their datasets use fsspec under the hood so this path can be an s3 bucket or a local path and it all just works.

Is this different from snakemake?

Some similar concepts, but Kedro is specialised on the ML engineering workflow and team collaboration

Hello maintainer here - hopefully some of our open source users chime in, but my favourite will always be NASA: https://github.com/nasa/ML-airport-configuration

Donation is the normal term for proejcts joining the foundation - in doing so you establish a steering committee where members are entitled to voting rights. To graduate as an incubation project, 5 organisations need to join your board and the project is thus the priorities of the project will no longer be driven by just one organisation.

In the short term there is still a full time internal team staffed and maintaining the project, but excitingly we now have a mechanism for new collaborators to properly come onboard.

I should clarify by short term - the foreseeable future is a more appropriate description

Am from McK not involved with Kedro, it’s been open source for awhile now and I don’t think we are abandoning it, but it was apparently donated to the Linux foundation.

As far as impressions of it as a tool go it’s really just an opinionated way to build a pipeline and structure a project. Which is pretty useful when you have many. If you’re doing something that you know will be one off it is a bit overkill. Like using airflow when a makefile will do. There are some pretty nice plug-ins though if you want to try and have an opinionated ecosystem.

Worked with McK on a modeling tool built with a data pipeline in Kedro - it's very opinionated as to how the pipeline has to be built, and I was not super convinced that it would be useful to use it in any other context (as in, outside of a collaboration with McK). Also, YMMV, but I have never heard of any company doing ML or data pipelines using Kedro on their own.

It’s a framework to organize data science code into pipelines.

E.g. if your default might be a series of Python files connected by a master script and some config files, Kedro organizes them into an explicit Pipeline object. Think loading, cleaning, feature gen, model training, mode predictions, etc.