Re:Flyte: A Cloud Native Machine Learning and Data Processin...
by matthewphsmith on Tuesday January 07, 2020 @07:55PM
(#21987116)
It is definitely possible to leverage Flyte's features here for sharing partial outputs. It does, however, require formatting your pipelines in a particular manner for this to be supported natively (i.e. create a task which computes a view or similar. Then that task is shared among different pipelines). It might be a little verbose, but, in my opinion, it is preferable because it modularizes pipelines into tasks which are individually tractable for testing and validation--especially as the tasks and pipelines evolve in a large organization! Further, the verbosity of authoring such pipelines can be reduced significantly by making good use of the flytekit library.
Additionally, there is always the option of introducing a custom plugin. Although it would take more effort up front, one can really let their imagination run wild and introduce behaviors as needed.
Additionally, there is always the option of introducing a custom plugin. Although it would take more effort up front, one can really let their imagination run wild and introduce behaviors as needed.