News for hackers. Stuff that matters.
Re:Introduction to Apache Airflow by simo7 on Friday May 29, 2020 @10:10AM (#23350128)
I would add these gotchas/recommendations:

- Airflow the ETL framework is quite bad. Just use Airflow the scheduler/orchestrator: delegate the actual data transformation to external services (serverless, kubernetes etc.).

- Don't use it for tasks that don't require idempotency (eg. a job that uses a bookmark).

- Don't use it for latency-sensitive jobs (this one should be obvious).

- Don't use sensors or cross-DAG dependencies.

So yeah unfortunately it's not a good fit for all the use cases, but it has the right set of features for some of the most common batch workloads.

Also python as the DAG configuration language was a very successful idea, maybe the most important contributor to Airflow success.

Re:Introduction to Apache Airflow by prions on Friday May 29, 2020 @11:19AM (#23350959)
> - Don't use it for tasks that don't require idempotency (eg. a job that uses a bookmark).

You can totally design your tasks to be idempotent - but its up to you to make them that way. The scheduler or executor doesn't have any context into your job.

This is why I encourage people to use a unified base operator and then pass their own docker containers to it. Aka like how https://medium.com/bluecore-engineering/were-all-using-airfl... outlines it.

> - Don't use it for latency-sensitive jobs (this one should be obvious).

IIRC this is being addressed in Airflow 2.0

> - Don't use sensors or cross-DAG dependencies.

This is a little extreme. I've never ran into issues with cross dag dependencies or sensors. They make managing my DAGs way easier because wee can separate computation dags from loading dags.

context: I built/manage my company's Airflow platform. Everything is managed on k8s.

Re:Introduction to Apache Airflow by simo7 on Friday May 29, 2020 @11:47AM (#23351379)
> You can totally design your tasks to be idempotent

Yes of course, I mean Airflow is not a good fit for the tasks you don't want to be idempotent (I think most but not all tasks should be idempotent).

> I've never ran into issues with cross dag dependencies

I believe Airflow docs advice against them when possible. I see why from my experience: less visibility and more complexity, especially for backfills.

Re:Introduction to Apache Airflow by domenp on Friday May 29, 2020 @03:02PM (#23354151)
> context: I built/manage my company's Airflow platform. Everything is managed on k8s

My team is running Airflow on a single node but we're slowly outgrowing this setup. We're considering running jobs on k8s.

Curious what's your setup like? Is your cluster of a fixed size or does it scale with the load?

Re:Introduction to Apache Airflow by ForHackernews on Friday May 29, 2020 @06:15PM (#23356180)
Using the KubernetesPodOperator for everything adds a huge amount of overhead. You still need Airflow worker nodes, but they're just babysitting the K8S pods doing the real work.

I know it's 2020 and memory is cheap or whatever, but Airflow is shockingly wasteful of system resources.

Re:Introduction to Apache Airflow by rywalker on Friday May 29, 2020 @12:59PM (#23352467)
re: ETL framework - you can get a lot done with the built-in Airflow Operators, including the PythonOperator (and bring in any python dependency you like) and BashOperator (call any CLI, etc.) - it's not drag-and-drop, but I've found it to be quite versitile.

re: idempotency - yes, make your workflow tasks idempotent.

re: latency - this is being worked on very actively. Ash (PMC member) has committed to working on task latency almost exclusively until it's resolved

re: sensors, there is some great work from Airbnb to improve: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+A...

Re:Introduction to Apache Airflow by hn2017 on Friday May 29, 2020 @11:13AM (#23350872)
If using a docker or kubernetes operator, you can't make use of operators or Airflow connections. Do you just work without those?
Re:Introduction to Apache Airflow by simo7 on Friday May 29, 2020 @11:35AM (#23351189)
Correct, I use operators only to delegate the actual workload to an external service.
Re:Introduction to Apache Airflow by hn2017 on Friday May 29, 2020 @06:01PM (#23356051)
Got it, to be clear - do you still use the Airflow for storing connections, or no? If no, how do you store your credentials? We've only done a POC and we've discovered a higher than expected learning curve.
Re:Introduction to Apache Airflow by IanCal on Friday May 29, 2020 @10:59AM (#23350678)
This is handy thanks - I deal mostly with luigi and this helps me place airflow a bit better.
Re:Introduction to Apache Airflow by contravariant on Friday May 29, 2020 @11:28AM (#23351105)
Sensors are quite helpful when you don't know the exact moment your data will come in.
Re:Introduction to Apache Airflow by simo7 on Friday May 29, 2020 @11:39AM (#23351267)
I think it's better to place everything in one DAG if that solves the problem. If it doesn't then sensors are ok I guess, but I would try to avoid them otherwise.
Re:Introduction to Apache Airflow by prions on Friday May 29, 2020 @11:45AM (#23351359)
I really disagree here. Monolithic dags are a bigger pain to manage.

Breaking them out into smaller dags makes retrying/backfilling/etc a lot more straightforward. It also lets you reuse those pieces more easily.

We have compute DAGs that are the upstream dependency for many other dags. Originally, this dag was monolithic and loaded data into one table. But because the dag is split into computation and loading we can easily add more downstream dags without changing how the first one operated.

Re:Introduction to Apache Airflow by bosie on Friday May 29, 2020 @10:49AM (#23350551)
What do you mean by bookmark in this context?
Re:Introduction to Apache Airflow by vorpalhex on Friday May 29, 2020 @10:59AM (#23350681)
"I have processed 5/15 records, next run I need to start at record 6". Bookmarking is a common concept for working through a large job in several small runs.
Re:Introduction to Apache Airflow by devonkim on Friday May 29, 2020 @11:25AM (#23351052)
I wonder if this difference in jargon has origins in different sub-fields. I recognize that concept as "checkpoints" across different companies but I also remember seeing the term from data science folks and thinking that I just don't know the concept.
Re:Introduction to Apache Airflow by nickpeterson on Friday May 29, 2020 @11:33AM (#23351155)
Often called a Waterline/Watermark as well.
Re:Introduction to Apache Airflow by sk5t on Friday May 29, 2020 @01:09PM (#23352593)
I'd call it a watermark too; "checkpoint" can mean an intermediate point in a multi-step database transaction.
Re:Introduction to Apache Airflow by jstarfish on Friday May 29, 2020 @01:09PM (#23352598)
I've always called them offsets.
Re:Introduction to Apache Airflow by wiml on Friday May 29, 2020 @04:44PM (#23355300)
I would call it "a read cursor (that's updated within the transaction)", but I'm boring.
Re:Introduction to Apache Airflow by unixhero on Monday June 01, 2020 @11:08AM (#23379301)
Cursor nomenclature is used for describing where in a sequence the operator is during a code execution pass in database query or similar right?

I think I kind of immediately understand "cursor" the best in this context. I also agree, it's a little boring and definitely old school:).