Worst of all, if you accept Bisquick, then you open the door to an explosion of Bisquick options. Its a combinatorial explosion of pre-mixed ingredients. In a dystopian future, perhaps people stop buying flour or salt, and the ONLY way you can make food is to buy the right kind of Bisquick. Might make a kind of mash up of a baking show and Black Mirror.
Anyway, yeah, Airflow (and so many other tools) feel like Bisquick. It has all the strengths, but also all the weaknesses, of that model.
Higher-order abstractions can be a productivity boon but have costs when you fight their paradigm or need to regularly interact with lower layers (in ways the designs didn't presume).
Airflow and similar tools are doing four things:
A) Centralized cron for distributed systems. If you don't have a unified runtime for your system, the old ways of using Unix cron, or a "job system" become complex because you don't have centralized management or clarity for when developers should use one given scheduling tool vs another.
B) Job state management. Job can fail and may need to be retried, people alerted, etc ... Most scheduling system has some way to do deal with failure too, but these tools are now treating this as stored state
C) DAGs, complex batch jobs are often composed of many stages with dependencies. And you need the state to track and retry stages independently (especially if they are costly)
D) What many of these tools also try to do, is tie the computation performing a given job to the scheduling tool. This now seems to be an antipattern. They also try to have "premade" job stages or "operators" for common tasks. These are a mix of wrappers to talk to different compute systems and actual compute mechanisms themselves.
If you have the kind of system that is either sufficiently distributed, or heterogeneous enough that you can't use existing schedulers, you need something with #A, but if you also need complex job management, you need #A, #B and #C, and having rebuilt my own my times, using a standard system is better when coordinating between many engineers. What seems necessary in general is #D.
Ed: I should say, I really like this metaphor, and I expect it will crop up in my thinking in the future.
Couscous, Bisquick and other low-or-no-heat, premixed, just-add-water solutions are a godsend to have when a tornado takes out your gas line or electric grid.
- Airflow the ETL framework is quite bad. Just use Airflow the scheduler/orchestrator: delegate the actual data transformation to external services (serverless, kubernetes etc.).
- Don't use it for tasks that don't require idempotency (eg. a job that uses a bookmark).
- Don't use it for latency-sensitive jobs (this one should be obvious).
- Don't use sensors or cross-DAG dependencies.
So yeah unfortunately it's not a good fit for all the use cases, but it has the right set of features for some of the most common batch workloads.
Also python as the DAG configuration language was a very successful idea, maybe the most important contributor to Airflow success.
You can totally design your tasks to be idempotent - but its up to you to make them that way. The scheduler or executor doesn't have any context into your job.
This is why I encourage people to use a unified base operator and then pass their own docker containers to it. Aka like how https://medium.com/bluecore-engineering/were-all-using-airfl... outlines it.
> - Don't use it for latency-sensitive jobs (this one should be obvious).
IIRC this is being addressed in Airflow 2.0
> - Don't use sensors or cross-DAG dependencies.
This is a little extreme. I've never ran into issues with cross dag dependencies or sensors. They make managing my DAGs way easier because wee can separate computation dags from loading dags.
context: I built/manage my company's Airflow platform. Everything is managed on k8s.
Yes of course, I mean Airflow is not a good fit for the tasks you don't want to be idempotent (I think most but not all tasks should be idempotent).
> I've never ran into issues with cross dag dependencies
I believe Airflow docs advice against them when possible. I see why from my experience: less visibility and more complexity, especially for backfills.
My team is running Airflow on a single node but we're slowly outgrowing this setup. We're considering running jobs on k8s.
Curious what's your setup like? Is your cluster of a fixed size or does it scale with the load?
I know it's 2020 and memory is cheap or whatever, but Airflow is shockingly wasteful of system resources.
re: idempotency - yes, make your workflow tasks idempotent.
re: latency - this is being worked on very actively. Ash (PMC member) has committed to working on task latency almost exclusively until it's resolved
re: sensors, there is some great work from Airbnb to improve: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+A...
Breaking them out into smaller dags makes retrying/backfilling/etc a lot more straightforward. It also lets you reuse those pieces more easily.
We have compute DAGs that are the upstream dependency for many other dags. Originally, this dag was monolithic and loaded data into one table. But because the dag is split into computation and loading we can easily add more downstream dags without changing how the first one operated.
I think I kind of immediately understand "cursor" the best in this context. I also agree, it's a little boring and definitely old school:).
For this specific use case, I use healthchecks.io - trivial to deploy in almost any context which can ping a public URL. Generous free tier limits so I've got that going for me which is nice.
The single best reason to use airflow is that you have some data source with a time-based axis that you want to transfer or process. For example, you might want to ingest daily web logs into a database. Or maybe you want weekly statistics generated on your database, etc.
The next best reason to use airflow is that you have a recurring job that you want not only to happen, but to track it's successes and failures. For example, maybe you want to garbage-collect some files on a remote server with spotty connectivity, and you want to be emailed if it fails for more than two days in a row.
Beyond those two, Airflow might be very useful, but you'll be shoehorning your use case into Airflow's capabilities.
Airflow is basically a distributed cron daemon with support for reruns and SLAs. If you're using Python for your tasks, it also includes a large collection of data abstraction layers such that Airflow can manage the named connections to the different sources, and you only have to code the transfer or transform rules.