Re:Lessons learned from running Apache Airflow at scale
by pharmakom on Monday May 23, 2022 @01:05PM
(#31481705)
This can be a much better approach than upgrading the DAG description language to a true programming language. It forces anything complex to happen at build time where it can do less damage. Plus, we can often use the same library to do static analysis on the output
[ Parent
| on HN ]
Re:Lessons learned from running Apache Airflow at scale
by savin-goyal on Monday May 23, 2022 @01:59PM
(#31482588)
Metaflow provides a similar concept to interface with Step Functions and Argo Workflows in Python - https://docs.metaflow.org/going-to-production-with-metaflow/...