News for hackers. Stuff that matters.
Re:Flyte: A Cloud Native Machine Learning and Data Processin... by matthewphsmith on Tuesday January 07, 2020 @07:55PM (#21987116)
It is definitely possible to leverage Flyte's features here for sharing partial outputs. It does, however, require formatting your pipelines in a particular manner for this to be supported natively (i.e. create a task which computes a view or similar. Then that task is shared among different pipelines). It might be a little verbose, but, in my opinion, it is preferable because it modularizes pipelines into tasks which are individually tractable for testing and validation--especially as the tasks and pipelines evolve in a large organization! Further, the verbosity of authoring such pipelines can be reduced significantly by making good use of the flytekit library.

Additionally, there is always the option of introducing a custom plugin. Although it would take more effort up front, one can really let their imagination run wild and introduce behaviors as needed.

Re:Flyte: A Cloud Native Machine Learning and Data Processin... by crorella on Tuesday January 07, 2020 @08:58PM (#21987496)
I see, thanks. I was hoping for this to be more automatic.