I do agree it makes sense fornscale, but if your data fits in memory Flyte native constructs shine. For example it will ensure your data is stored / serialized correctly. Allows you to use polars, vaex, duckdb etc. tbh I am a huge proponent of vertical scaling till you can get the mileage.

It also supports - gpu allocation, spot instances and collaboration across multiple users. I do not think it is a wrong choice if you feel your complexity will grow.

PS. I am a maintainer of the project