LLM Workflows then Agents: Getting Started with Apache Airflow

Re:LLM Workflows then Agents: Getting Started with Apache Ai... by gre on Monday March 31, 2025 @08:59PM (#43541741)

Here's my problems with MWAA (amazon hosted airflow.) I have about 100 dags which maxes out the scheduler thread. Airflow parses all the files every minute so it's always parsing around 94% cpu. I could run a second scheduler thread if I coordinate with my SRE team and get the terraform deployed...it's really tedious.

Related possibly, my dags get kill -9 for no apparent reason. The RAM usage is not that high, maybe 2gb out of 8gb system RAM in use. No reason is given in the logs.

I am trying to switch to dagster, not because it's awesome, but because it hasn't crashed randomly on me.

[ Parent | on HN ]

Re:LLM Workflows then Agents: Getting Started with Apache Ai... by mblast311 on Monday March 31, 2025 @11:04PM (#43542400)

MWAA is hot garbage. I had similar issues and switched to running it on EKS instead.

[ Parent | on HN ]

Re:LLM Workflows then Agents: Getting Started with Apache Ai... by alittletooraph2 on Tuesday April 01, 2025 @09:54AM (#43546823)

This feels like an MWAA issue but I understand how that often gets conflated with it being an Airflow issue.

[ Parent | on HN ]

Re:LLM Workflows then Agents: Getting Started with Apache Ai... by gre on Tuesday April 01, 2025 @11:00AM (#43547605)

You're right, it doesn't happen when developing locally, only in MWAA. This was the answer given by the Airflow team as well and I figured they would punt before I asked.

I realize Amazon is taking an open source project and making a ton of money on it (the instance prices are ridiculous for what you get) and the incentives are misaligned for the Airflow team to help AWS make it better unless AWS paid them to help fix it.

It's crap all around, and Airflow gets a bad rap from AWS's terrible MWAA product based on it.

[ Parent | on HN ]