Data Engineering Patterns with Apache Airflow [video]

Re:Data Engineering Patterns with Apache Airflow [video] by tpaschalis on Wednesday August 29, 2018 @07:05PM (#17872861)

(Newbie Airflow user here). I believe one easy way to do it is by using Airflow's 'sensors'.

Sensors are operators which poke continuously with an action until it returns True (eg. until a file exists, an API gives a specific response, a process/query has finished).

Another way to do it would be to 'XComs', small pieces of information flying between DAGs, or 'Triggers', but these require some more setup, and IMO depend more on the way you're setting up your tasks.

[ Parent | on HN ]

Re:Data Engineering Patterns with Apache Airflow [video] by xkcd-sucks on Wednesday August 29, 2018 @08:46PM (#17873297)

Yeah the issue is once a sensor fires once, it doesn't reset and keep firing on new data

My homegrown solution is a sensor at the beginning, and at the end an airflow api call to trigger a dag run of the same dag. Not DagRunOperator because then no dag would never finish due to infinite recursion

It seems kinda sketchy so I'm considering a lower level Celery implementation or even GenStage

[ Parent | on HN ]