Re:New Apache Airflow Operators for Google Generative AI
by annexrichmond on Monday August 12, 2024 @04:00PM
(#41228652)
Airflow owning the scheduling, retry, task branching/dependencies is fine. But a few issues: to take advantage of Airflow's core features (XCOMs, task mapping, etc) your biz logic/tasks must be in the same venv, which is not sustainable as teams grow. Once you have external tasks, you now have a more complex system to operate and test, and your Airflow installation is now very overkill (you need a pool of workers just to monitor external... workers?).
[ Parent
| on HN ]
Re:New Apache Airflow Operators for Google Generative AI
by notatallshaw on Tuesday August 13, 2024 @11:32AM
(#41236438)
You don't need to have your biz tasks in the same venv as your airflow worker to take advantage of task mapping or XCOM.
You just need a well defined interface on how to call your biz tasks, and then you can use any of ExternalPythonOperator, DockerOperator, DockerSwarmOperator, KubernetesPodOperator, etc. etc. or write your own to pass in values or data to your task however you want.
Airflow is quite complex, and I don't recommend it as people's go to, but IMO that's in large part because it is so unopinionated about how you call and run your tasks and leaves the configuration up to you. But this also means it ends up being a lot of people's choice because they are able to get it to fit what they need.