News for hackers. Stuff that matters.
Re:Airflow's Problem by stkbailey on Thursday August 04, 2022 @08:51AM (#32342165)
We didn't look into a dataflow engine specifically, in part because we have a heterogeneous set of workfloads. Our core use case is loading mission critical data in chunks, but it is also coordinating SaaS tools and managed services like Sagemaker. So the sort of "just run this arbitrary code" reliably and scalably is an important role in our case, not just the dataflow part of things.

There are no comments on this comment yet.