One way to achieve that is to use externally triggered DAGs [1][2].

Instead of trying to make the Airflow scheduler work as event-driven though, I've more tried to adapt my approach.

I've used Airflow as a bridge between a micro batch streaming system (Sparking Streaming --> S3, Airflow: S3 --> ...) successfully, but there are some challenges around the edge cases like files landing later than expected.

One approach you can take is to have the Airflow sensor hit a file that's a proxy for all of the files being present. For example, if your process could write hundreds of S3 files, once it's finished the last write for that hour (even if that happens late for whatever reason), then it could write a top-level OK file that the sensor hits.

Can you elaborate on your specific use case?

[1]: https://stackoverflow.com/a/50592573/149428

[2]: https://airflow.readthedocs.io/en/stable/scheduler.html#exte...

For "event-driven" scheduler, you may want to have a streaming system like Flink or Spark Streaming instead. You can schedule the startup of these pipelines with Airflow (e.g., using BashOperator), and handle the actual streaming with the streaming systems directly.

We use Airflow with this pattern extensively.

(Newbie Airflow user here). I believe one easy way to do it is by using Airflow's 'sensors'.

Sensors are operators which poke continuously with an action until it returns True (eg. until a file exists, an API gives a specific response, a process/query has finished).

Another way to do it would be to 'XComs', small pieces of information flying between DAGs, or 'Triggers', but these require some more setup, and IMO depend more on the way you're setting up your tasks.

Yeah the issue is once a sensor fires once, it doesn't reset and keep firing on new data

My homegrown solution is a sensor at the beginning, and at the end an airflow api call to trigger a dag run of the same dag. Not DagRunOperator because then no dag would never finish due to infinite recursion

It seems kinda sketchy so I'm considering a lower level Celery implementation or even GenStage