Re:Kedro – Creating reproducible, maintainable and modular d...
by barefeg on Saturday January 22, 2022 @03:47PM
(#30040676)
Could you point me to the component responsible for storing the intermediate data of the DAG run? I was looking for this but couldn’t find it from quickly scanning the docs
Re:Kedro – Creating reproducible, maintainable and modular d...
by waylonwalker on Sunday January 23, 2022 @09:54AM
(#30046534)
By adding a catalog entry for each of your datasets with a filepath argument, it will save them to that place. All their datasets use fsspec under the hood so this path can be an s3 bucket or a local path and it all just works.
Unlike similar projects kedro is just a python framework. It lets me build, deploy, and orchestrate however works best for my team.