yep some of these ML platform tools only make sense once you reach a certain scale/set of problems. For flyte, it was created because Lyft had a very heterogeneous set of tools one could build a ML pipeline with. So they made it really easy to integrate and importantly serialize, data between systems. E.g. sql -> python -> spark -> python. But if all your data fits in memory, or you only one system, you might not be a great fit for Flyte.

What I try to do, is understand who created the platform, and understand the environment of that company. Then that will give you a better idea of whether it makes sense for you or not.

I do agree it makes sense fornscale, but if your data fits in memory Flyte native constructs shine. For example it will ensure your data is stored / serialized correctly. Allows you to use polars, vaex, duckdb etc. tbh I am a huge proponent of vertical scaling till you can get the mileage.

It also supports - gpu allocation, spot instances and collaboration across multiple users. I do not think it is a wrong choice if you feel your complexity will grow.

PS. I am a maintainer of the project

Yep - it's on my TODO to get https://github.com/flyteorg/flyte/issues/2627 over the line!

Thank you for sharing. If you do not have platform engineers look at union.ai. They offer a manager version of Flyte