Lessons learned from running Apache Airflow at scale

Re:Lessons learned from running Apache Airflow at scale by vvladymyrov on Monday May 23, 2022 @11:56AM (#31480654)

Airflow brought one of the best tools with nice UI for running pipelines back in 2014-2016. But now days engineers should be aware about easier to use options and don’t choose Airflow blindly as default choice. IMHO for 80-90% of cases orchestration system should not use code at all - it should be DAGs as a config code. Airflow is popular and teams keep choosing it for building simple DAGs and incurring avoidable otherwise Airflow maintenance costs.

Databricks Orchestration pipeline, AWS Step Functions - good examples of DAGs as a configuration.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by avemg on Monday May 23, 2022 @12:16PM (#31480961)

I've used AWS Step Functions extensively over the past several years and give me code every day of the week over the Stepfunctions json config. Once you get beyond a few simple steps, it gets very hard to look at the config and understand what's going on with it. Especially true when you haven't look at the config in awhile. The DAG visualizer definitely helps, but as soon as things get beyond the trivial I long for a different tool.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by TYPE_FASTER on Monday May 23, 2022 @12:37PM (#31481276)

AWS offers a service for managed Airflow: https://aws.amazon.com/managed-workflows-for-apache-airflow/

Makes me wonder if Amazon internally was using Step Functions, ran into issues trying to scale to larger graphs, realized multiple teams were using Airflow, and created the Managed Airflow service.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by latchkey on Monday May 23, 2022 @12:27PM (#31481132)

Why not just model the json as objects in (insert favorite language) and then use that code to generate the json?

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by entropicdrifter on Monday May 23, 2022 @12:33PM (#31481212)

Ah yes, a home-made framework to generate configurations for your framework that's supposed to make your life easier. That way you can maintain your code that maintains your configs that make it easier to run your code that you have to maintain!

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by latchkey on Monday May 23, 2022 @02:02PM (#31482639)

Actually, yes. It allows for easier unit and integration testing as well. The original complaint is that things were getting hard to read and they wished there was code for this. It seems logical to create a framework for the json configuration files so that they can be easily mocked and tested. As someone who greatly values spending time on automated testing, it seems weird to not think of it this way.

Quick google shows that others have done things like this already...

[1] https://noise.getoto.net/2021/10/14/using-jsonpath-effective...

[2] https://aws.amazon.com/about-aws/whats-new/2022/01/aws-step-...

[3] https://docs.aws.amazon.com/step-functions/latest/dg/sfn-loc...

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by pharmakom on Monday May 23, 2022 @01:05PM (#31481705)

This can be a much better approach than upgrading the DAG description language to a true programming language. It forces anything complex to happen at build time where it can do less damage. Plus, we can often use the same library to do static analysis on the output

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by savin-goyal on Monday May 23, 2022 @01:59PM (#31482588)

Metaflow provides a similar concept to interface with Step Functions and Argo Workflows in Python - https://docs.metaflow.org/going-to-production-with-metaflow/...

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by coolsunglasses on Monday May 23, 2022 @12:24PM (#31481092)

I was until a week or two ago part of a team that build datasets with extensive dependencies (thus, complicated DAGs)

v1 of the system built before I joined was Step Functions and the like. It gets hairy just as you say.

v2 I built and designed with the lead data engineer, we called it Coriaria originally. We're hoping/planning to open source it eventually, although it's a little wrapped around our company's internal needs & systems.

It chooses neither "config" strictly speaking nor "code" for the DAG, instead the primary representation/state is all in the PostgreSQL database which tracks the dataset dependencies and how each dataset is built. It's a DAG in PostgreSQL as well.

To make dataset creation and management easier, I also wrote a custom Terraform provider for Coriaria. This made migrating datasets into the new system dramatically faster. The provider is really nice, supports `terraform import` and all that. Currently we have it setup so that there are separate roles/accounts that can modify an existing dataset, but reading state only requires authentication, not authorization. This enables one team to depend on another team's dataset as an upstream data source for their datasets without granting permission to modify it or create a potentially stale copy of the dataset. Terraform's internal DAG representation of the resource dependencies is leveraged because "parent_datasets" references the upstream datasets directly, including the ones we don't build.

We're able to depend on datasets we don't build ourselves because the system has support for Glue catalog backends to track and register partition availability.

Currently, it builds most of the datasets using AWS Athena & S3, however this is abstracted over a single step function. There's no DAG of step functions, it's just a convenient wrapper for the Athena query execution.

The system also explicitly understands dataset builds and validations as separate steps. The dashboard makes it easy to trace the DAG and see which datasets are blocking a dataset build.

We're adding more integrations to it soon so that other ways of kicking off dataset builds and validations are available.

If people are interested in this I can begin lobbying for open sourcing the system. My colleague wanted to open source it as well.

All else fails, I'll rebuild it from scratch because I don't like the existing solutions for managing datasets. We've been calling it a data-flow orchestration system or ETL orchestration system, not sure what would be most meaningful to people.

I think the main caveat to this system is that I'm not sure how much use it'd be for streaming data pipelines, but it could manage the discretization of streaming into validated partitions wherever streamed data is sunk into. Our operating assumptions are that you want validated datasets to drive business decisions, not raw event data streamed in from Kafka. Making sure the right data is located in each daily (or hourly) partition is part of that validation.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by carschno on Monday May 23, 2022 @12:11PM (#31480876)

Do you have more examples for better tools, ideally open source (unlike AWS Step functions)?

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by gadflyinyoureye on Monday May 23, 2022 @01:08PM (#31481757)

Flowable is a BPNM system. You can do a lot of async calls with it. https://www.flowable.com/open-source

We use it for a complex pricing process that invokes 30-40 micro services that can take up to minutes per step.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by qw on Monday May 23, 2022 @01:29PM (#31482068)

Kamelets and the Karavan UI in combination with k8s and Knative for "serverless" integrations looks interesting.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by jonpon on Monday May 23, 2022 @03:38PM (#31483928)

We at magniv.io are building an alternative.

Our core is open source https://github.com/MagnivOrg/magniv-core

We can set you up with our hosted if you would like to poke around!

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by antupis on Monday May 23, 2022 @02:28PM (#31483094)

We are using prefect+dbt and I like it. Altought they are doing huge rewrite at the moment.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by nomilk on Monday May 23, 2022 @12:06PM (#31480807)

> teams keep choosing it for building simple DAGs

I am part of one such team. We were using Windows Task Scheduler on a windows VM to run jobs, we figured it would be a nice idea to (dramatically) modernise and move to airflow, but we grossly underestimated the complexity, learning curve, and surrounding tools it requires. In the end we (data science team) didn't get a single production task up and running. The data engineers had much more success with it thought, probably because they dedicated much more time to it.

Will look forward to trying AWS Step Functions.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by commandlinefan on Monday May 23, 2022 @12:10PM (#31480866)

I tried installing Airflow locally to just play around with it and make sense of what it's good for and finally gave up after a few days - the install alone is insanely complicated, with lots of tricky hidden dependencies.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by always_left on Monday May 23, 2022 @12:18PM (#31481008)

Did you try installing with docker? You would just download docker, `docker-compose up --build` and you'll be good to go locally (usually)

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by idleprocess on Monday May 23, 2022 @01:00PM (#31481648)

I can second this. We were up-and-running with Docker on our dev machines in just a few minutes. A native installation involves substantially more setup (Python, databases, Redis and/or Rabbit, etc.). The published docker-compose file will handle all of that for you. We have a very small data engineering team and have been able to move very quickly with Docker and AWS ECS (for orchestrating containers in test and prod environments).

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by flowair on Wednesday May 25, 2022 @10:34AM (#31505143)

download the astronomer CLI and it will spin up a local dockerized instance of airflow in about 1 minute tops https://docs.astronomer.io/astro/install-cli

[ Parent | on HN ]