News for hackers. Stuff that matters.

A comment on Lessons learned from running Apache Airflow at scale.

Re:Lessons learned from running Apache Airflow at scale by dekhn on Monday May 23, 2022 @12:41PM (#31481332)
I tried to run airflow; I found pretty much everything about it to be wrong for my usecase. Why can't I easily upload a workflow through the UI? Why doesn't it handle S3 file staging for me?
[ Parent | on HN ]
Re:Lessons learned from running Apache Airflow at scale by hatware on Monday May 23, 2022 @12:48PM (#31481445)
It definitely takes some time getting used to the quirks of Airflow. I know it took 6 months of running it at my last gig to really understand what was happening underneath the UI.

With great control comes great responsibility.

[ Parent | on HN ]
Re:Lessons learned from running Apache Airflow at scale by dekhn on Monday May 23, 2022 @12:51PM (#31481483)
actually I concluded it was just not that great a workflow engine. It's probably just intended for a different use case than mine.
[ Parent | on HN ]
Re:Lessons learned from running Apache Airflow at scale by jonpon on Monday May 23, 2022 @03:44PM (#31483994)
Would love to hear more about your use case and your issues -- you can sign up on our website (magniv) or send me an email jon at our domain.
[ Parent | on HN ]