matthewphsmith

Thanks for the link! Looking at Prefect's comparison to Airflow (https://docs.prefect.io/core/welcome/why_not_airflow.html#wh...), I think it sums up a lot of the same areas we tried to address with Flyte! Particularly the bullet-points in the 'Overview' section were all things, to the word, that were paramount for us to solve at Lyft. Now perhaps we need to publish an article on comparison to Prefect ;)

It is definitely possible to leverage Flyte's features here for sharing partial outputs. It does, however, require formatting your pipelines in a particular manner for this to be supported natively (i.e. create a task which computes a view or similar. Then that task is shared among different pipelines). It might be a little verbose, but, in my opinion, it is preferable because it modularizes pipelines into tasks which are individually tractable for testing and validation--especially as the tasks and pipelines evolve in a large organization! Further, the verbosity of authoring such pipelines can be reduced significantly by making good use of the flytekit library.

Additionally, there is always the option of introducing a custom plugin. Although it would take more effort up front, one can really let their imagination run wild and introduce behaviors as needed.

That is something to which the SDK was designed to be extended--in fact, the SDK was designed specifically to be extended to different programming models besides the rudimentary one we provide out-of-box. However, we have not had the time to take on the implementation ourselves as the existing one serves our purposes at Lyft well and we are but a small team. So we do hope that someone in the community with a knack for syntax can help improve things!