News for hackers. Stuff that matters.

A comment on Flyte: A Cloud Native Machine Learning and Data Processing Platform. Parent comment

Re:Flyte: A Cloud Native Machine Learning and Data Processin... by zerovar on Tuesday January 07, 2020 @08:56PM (#21987480)
Have Lyft migrated all their workflows from Airflow to Flyte? Or does Airflow still play a role alongside Flyte? Was assuming Lyft is running workflows in Airflow from this post https://eng.lyft.com/running-apache-airflow-at-lyft-6e53bb8f...
[ Parent | on HN ]
Re:Flyte: A Cloud Native Machine Learning and Data Processin... by kumare3 on Tuesday January 07, 2020 @10:53PM (#21988126)
Another great question. So Airflow is used at Lyft for ETL. I think for traditional ETL it still is a good fit. But, there is an effort to not just migrate, but rethink how we can leverage Flyte's capabilities to improve our ETL experience.

But, as it exists, we have a FlyteAirflowOperator, so that users can easily connect their Airflow pipelines with Flyte and write the new ones on Flyte alone.

Stay tuned for developments on this front :)

[ Parent | on HN ]