News for hackers. Stuff that matters.

A comment on Lessons learned from running Apache Airflow at scale. Parent comment

Re:Lessons learned from running Apache Airflow at scale by rockostrich on Monday May 23, 2022 @02:27PM (#31483084)
We had a similar DAG that was the result of migration a single daily Luigi pipeline to Airflow. I started identifying isolated branches and breaking them off with external task sensors back to the main DAG. This worked but it's a pain in the ass. My coworker ended up exporting the graph to graphviz and started identifying clusters of related tasks that way.
[ Parent | on HN ]
Re:Lessons learned from running Apache Airflow at scale by mywittyname on Monday May 23, 2022 @02:41PM (#31483281)
I've not had the best luck with ExternalTaskSensors. There have been some odd errors like execution failing at 22:00:00 every day (despite the external task running fine).
[ Parent | on HN ]