Kedro – Creating reproducible, maintainable and modular data science code

Re:Kedro – Creating reproducible, maintainable and modular d... by barefeg on Saturday January 22, 2022 @03:47PM (#30040676)

Could you point me to the component responsible for storing the intermediate data of the DAG run? I was looking for this but couldn’t find it from quickly scanning the docs

[ Parent | on HN ]

Re:Kedro – Creating reproducible, maintainable and modular d... by joelschw on Saturday January 22, 2022 @04:04PM (#30040816)

Here you go - https://kedro.readthedocs.io/en/0.17.6/09_development/03_com...

[ Parent | on HN ]

Re:Kedro – Creating reproducible, maintainable and modular d... by waylonwalker on Sunday January 23, 2022 @09:54AM (#30046534)

By adding a catalog entry for each of your datasets with a filepath argument, it will save them to that place. All their datasets use fsspec under the hood so this path can be an s3 bucket or a local path and it all just works.

[ Parent | on HN ]