News for hackers. Stuff that matters.
Re:Lessons learned from running Apache Airflow at scale by mywittyname on Monday May 23, 2022 @02:51PM (#31483416)
Lambdas have resource constraints that Airflow DAGs don't. Most notably, Airflow DAGs can run for any arbitrary length of time. And the local storage attached to the Airflow cluster is actual disk space, and not just a fake in-memory disk, making it possible process files larger than the amount of memory allocated to the DAG.

There's certainly some functionality overlap, but I don't see Lambda and Airflow as competitors. Each has capabilities that the other doesn't.

Re:Lessons learned from running Apache Airflow at scale by xtracto on Monday May 23, 2022 @03:20PM (#31483736)
I remember reading that you can attach EFS to lambdas. That would solve some of the storage issues.