kumare3

May I recommend looking into flyte.org, it is open source kubernetes native "orchestration" style tool, but essentially and infrastructure component that is geared to making your ML Engineers and Data scientists more productive. I think iteration velocity, dynamic infrastructure management and trackability are really important and fundamentally different needs of such products.

PS. I am a maintainer at flyte.org (thoughts are my own)

Thank you for sharing. If you do not have platform engineers look at union.ai. They offer a manager version of Flyte

I do agree it makes sense fornscale, but if your data fits in memory Flyte native constructs shine. For example it will ensure your data is stored / serialized correctly. Allows you to use polars, vaex, duckdb etc. tbh I am a huge proponent of vertical scaling till you can get the mileage.

It also supports - gpu allocation, spot instances and collaboration across multiple users. I do not think it is a wrong choice if you feel your complexity will grow.

PS. I am a maintainer of the project

Checkout Flyte.org and it’s sibling project https://www.union.ai/unionml

@tomrod, thank you for the callout. By the way we are integrating mlflow into Flyte in a way that you do not need to start the web server to view the logs. They are available Locally and statically in Flyte Ui. Ofcourse you cal also use mlflow server

Some of these were the core problems that we wanted to address as part of https://flyte.org. We started with a team first and multi-tenant approach at the core. For example, each team can have separate IAM roles, secrets are restricted to teams, tasks and workflows are shareable across teams, without making libraries. and it is possible to trigger workflows across teams. Each teams workflows are tasks and grouped using a construct called projects. It is even possible to separate execution clusters per team, per workflow onto separate k8s clusters. Also the platform is built to be managed and easily deployed.

I remember having this feeling a few years ago. What I realized is that airflow has taught us a few bad habits and also brought ahead an interesting paradigm of the vertical workflow engine.

I agree airflow is old, legacy and ideally folks should not use it, reality is there is a lot of pipelines already built with it - sadly. I think as a community we have to start moving away from it for more complicated problems.

Disclaimer: I created Flyte.org and heavily believe in decentralized development of DAGs and centralized management of infrastructure

This is cool, you could provide resilience to Red Engine, by providing it a backend using Flyte.org. Checkout example of making a new API on top of Flyte which has similarish feel - https://unionml.readthedocs.io/en/latest/index.html#quicksta....

Thus users could continue using RED, and if they want to scale to multiple machines or want resilience, you could allow them to switch out the backend to Flyte.

Disclosure: I am maintainer of Flyte. This is just a suggestion. Great work!

+1

Thank you. So at Lyft, there exists an AirflowFlyteOperator. It was designed to interop with Airflow. I know the community is working on open sourcing it. Keep a lookout for the same

Ohh you are welcome, join the slack channel and ask for help. The community is growing everyday - here are some examples of using it in python https://flytecookbook.readthedocs.io/en/latest/

Have you tried Flyte.org?

Spotify is moving to https://Flyte.org and building a Luigi to Flyte compiler. Stay tuned

Have you tried Flyte - flyte.org?

Roberto, this is absolutely one of our goals. When we started, it was with python2.7 still around. We would love contributions, ofcourse we will work with you and adapt it

Another great question. So Airflow is used at Lyft for ETL. I think for traditional ETL it still is a good fit. But, there is an effort to not just migrate, but rethink how we can leverage Flyte's capabilities to improve our ETL experience.

But, as it exists, we have a FlyteAirflowOperator, so that users can easily connect their Airflow pipelines with Flyte and write the new ones on Flyte alone.

Stay tuned for developments on this front :)

Great question, I am working on a follow up blog that will explain the differences in more detail. Flyte does take some inspiration from airflow, but it has a lot of important differences - Flyte natively understands data flow between tasks. This is achieved using its own type system created in protobuf - Flyte tasks are first class citizens and hence can be shared, reused and are always associated with an interface declaration - Flyte is container and kibernetes native. It is also multi tenant. - Flyte corn scheduler, control plane api and the actual execution engine are decoupled. Each workflow can be independently executed on a different execution engine - Flyte workflows are purely specification - defined in protobuf and Flyte tasks also - Flyte provides an event stream of the execution - since Flyte is aware of the data, it comes with built in memorization and auto cataloging - like airflow Flyte can have plugins in python, but it supports a richer plugin interface - Flyte is written in Golang and on top of kuberenetes It is definitely less mature in the open source, so please help us make it better. But it has been battle tested at Lyft for more than 3 years in production.