Lessons learned from running Apache Airflow at scale

Re:Lessons learned from running Apache Airflow at scale by thefourthchime on Monday May 23, 2022 @03:56PM (#31484096)

> AWS step functions are really the best in orchestration.

At our company, AWS Step is a disaster. You're effectively writing code in JSON/YAML. Anything beyond very simple steps becomes 2 pages of YAML that's very hard to read or write. There is no way to debug, polling is mostly unusable. Changes need to be deployed with CF which can take forever, or worse hang.

It's the most one of the most annoying technologies I've used in my 20+ years of engineering.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by quantisan on Monday May 23, 2022 @09:41PM (#31487081)

I feel you. That's why we wrote a little library on top of SFN so that we can program SFN with Clojure instead of YAML https://github.com/Motiva-AI/stepwise. Application code sits with SFN definition and SFN Tasks are automatically integrated as polling Activities from Clojure code.

Thoughtworks made a case for this distinction in https://martinfowler.com/articles/cant-buy-integration.html#...

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by ledauphin on Monday May 23, 2022 @08:12PM (#31486464)

yeah, to me the idea that anyone would want to write orchestration in something that isn't a programming language where you can write tests in the same language is just so backwards.

[ Parent | on HN ]