Flyte: A Cloud Native Machine Learning and Data Processing Platform

Re:Flyte: A Cloud Native Machine Learning and Data Processin... by tedmiston on Tuesday January 07, 2020 @09:10PM (#21987582)

It definitely feels like there is Airflow inspiration though poking through the site and docs, it seems that the devil is in the details with respect to the differences between the two... kinda like the relationship between Airflow and Prefect [1]. It looks like the barrier to entry is higher with Flyte but that there are benefits as well.

One example I see is that one can run Airflow simply without containers if desired with just simple Python functions whereas Flyte seems to be much more concerned with managing the execution environment for you (pros/cons to that).

Flyte also seems to be more "Kubernetes native" by default [2][3] vs with Airflow this is more of a choice amongst several executors.

I'd be curious to see a performance benchmark using comparable workflows vs Airflow with the Kubernetes Executor or Kubernetes Operator.

[1]: https://www.prefect.io/

[2]: https://kccncna19.sched.com/event/UaYY/flyte-cloud-native-ma...

[3]: https://www.youtube.com/watch?v=KdUJGSP1h9U

[ Parent | on HN ]

Re:Flyte: A Cloud Native Machine Learning and Data Processin... by matthewphsmith on Tuesday January 07, 2020 @09:33PM (#21987731)

Thanks for the link! Looking at Prefect's comparison to Airflow (https://docs.prefect.io/core/welcome/why_not_airflow.html#wh...), I think it sums up a lot of the same areas we tried to address with Flyte! Particularly the bullet-points in the 'Overview' section were all things, to the word, that were paramount for us to solve at Lyft. Now perhaps we need to publish an article on comparison to Prefect ;)

[ Parent | on HN ]

Re:Flyte: A Cloud Native Machine Learning and Data Processin... by tedmiston on Tuesday January 07, 2020 @09:40PM (#21987771)

Thank you for replying. I would definitely look forward to that post.

[ Parent | on HN ]