You can totally design your tasks to be idempotent - but its up to you to make them that way. The scheduler or executor doesn't have any context into your job.
This is why I encourage people to use a unified base operator and then pass their own docker containers to it. Aka like how https://medium.com/bluecore-engineering/were-all-using-airfl... outlines it.
> - Don't use it for latency-sensitive jobs (this one should be obvious).
IIRC this is being addressed in Airflow 2.0
> - Don't use sensors or cross-DAG dependencies.
This is a little extreme. I've never ran into issues with cross dag dependencies or sensors. They make managing my DAGs way easier because wee can separate computation dags from loading dags.
context: I built/manage my company's Airflow platform. Everything is managed on k8s.
Yes of course, I mean Airflow is not a good fit for the tasks you don't want to be idempotent (I think most but not all tasks should be idempotent).
> I've never ran into issues with cross dag dependencies
I believe Airflow docs advice against them when possible. I see why from my experience: less visibility and more complexity, especially for backfills.
My team is running Airflow on a single node but we're slowly outgrowing this setup. We're considering running jobs on k8s.
Curious what's your setup like? Is your cluster of a fixed size or does it scale with the load?
I know it's 2020 and memory is cheap or whatever, but Airflow is shockingly wasteful of system resources.
re: idempotency - yes, make your workflow tasks idempotent.
re: latency - this is being worked on very actively. Ash (PMC member) has committed to working on task latency almost exclusively until it's resolved
re: sensors, there is some great work from Airbnb to improve: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+A...
Breaking them out into smaller dags makes retrying/backfilling/etc a lot more straightforward. It also lets you reuse those pieces more easily.
We have compute DAGs that are the upstream dependency for many other dags. Originally, this dag was monolithic and loaded data into one table. But because the dag is split into computation and loading we can easily add more downstream dags without changing how the first one operated.
I think I kind of immediately understand "cursor" the best in this context. I also agree, it's a little boring and definitely old school:).
- Airflow the ETL framework is quite bad. Just use Airflow the scheduler/orchestrator: delegate the actual data transformation to external services (serverless, kubernetes etc.).
- Don't use it for tasks that don't require idempotency (eg. a job that uses a bookmark).
- Don't use it for latency-sensitive jobs (this one should be obvious).
- Don't use sensors or cross-DAG dependencies.
So yeah unfortunately it's not a good fit for all the use cases, but it has the right set of features for some of the most common batch workloads.
Also python as the DAG configuration language was a very successful idea, maybe the most important contributor to Airflow success.