dimberman

In 2004, Mark Zuckerberg called us "dumb fucks" for giving him our addresses and photos. Now we're giving model providers our math theorems, pharmaceutical research, and startup business plans.

Astronomer.io | (multiple roles) Fully remote-first compnay.

We maintain and offer Apache Airflow as a service to customers ranging from early stage start ups to Fortune 500s. We're hiring across the board. Front-end, python/data engs, and k8s/cloud experts.

I've worked for this company for two years now and it's been one of the funnest rides of my life. The culture is incredible, the people are incredibly smart yet humble, and the OSS Apache Airflow project has been exploding in popularity.

Please feel free to reach out if you have any interest or questions daniel [at] astronomer.io or you can apply on our site https://careers.astronomer.io/

Hi @theptip,

I'm an Airflow PMC and would love to know a bit more about your comparison :).

1. Have you tried Airflow 2.0? We made some pretty big overhauls both in terms of UI and backend. 2. DAG versioning is currently problematic, but DAG versioning is a "when" and not an "if" so should be in a future 2.x version :). That said could you describe a bit more about your deployment issues? User stories like this help us improve the product. 3. Have you looked into using KEDA with the CeleryExecutor? You could create KEDA queues for a lot of commonly used workflows and then you'd only need to use the python or bash operator to run those tasks instead of k8spodop. 4. Are you using the Airflow helm chart or did you custom roll a deployment?

Any feedback would be highly appreciated and I'm also glad to answer any questions you might have!

I'd say it's simpler than that.

Airflow is a backend project built by backend engineers.

Most UI people don't use Airflow or know what it is.

@ryanhamilton is the first front-end dev to become a committer on the project and that JUST happened a few months ago.

We're hiring multiple UI experts to rebuild the Airflow UI from scratch in react, so this is just the beginning :).

Hi didip!

could you make an airflow issue related to that or start a thread in the dev list? That could be interesting! (though you might want to wait until after the holidays as we're all a bit wiped :) )

On the OSS side we have a helm chart that is heavily based on the one we use at Astronomer. That should hopefully get you started (or you can reach out to astro and someone will help out with a demo if you want help on that)

I'm sure they will eventually, but Astronomer and (I believe) Cloud Composer have day 1 support.

Airflow 2.0 :)

WAIT. Sorry I misspoke.

The only thing that is no longer allowed is using a bitshift operator between a DAG and a task.

  task_1 >> task_2

is totally fine

  my_dag >> task_1

Is no longer allowed. Most of your DAG should be completely fine. Apologies for the miscommunication.

I'm sure they will, I'm not sure on their timeline though.

So there are a few options for that if you're interested!

1. If you're using the KubernetesExecutor, you can point to custom images for individual tasks, this will primarily work if you're storing DAGs in git or a volume (or if you want to handle baking in DAGs for different images).

2. You can use custom images in KEDA queues. This way you can simply point to a queue for all tasks in that DAG and they will run in that environment.

3. You can use the k8spodoperator. Now that the k8spodoperator allows for templating, it would be pretty easy to create a template for a pod and just inject different commands for different steps.

Hope that helps!

Unfortunately that is one feature that we had to take out, but you should check out the TaskFlow API, it's a very worthwhile trade-off!

Edit:

Sorry I misspoke here

The only thing that is no longer allowed is using a bitshift operator between a DAG and a task.

  task_1 >> task_2

is totally fine

  my_dag >> task_1

Is no longer allowed. Apologies for the miscommunication.

Thank you for the feedback! I'm gonna pass that on to some AWS experts in the community.

One really nice feature of 2.0 is now the "providers (hooks, operators, etc.) are released separately from Airflow itself. So you won't need to upgrade airflow to get improved AWS operators unless there is a breaking change.

A sensor would also work here. Especially with the new SmartSensor feature, sensors are basically free so you can set them up for event-based DAG executions.

We didn't separate task instances from timestamps YET purely because there was already so much to release that we didn't want to add more potential for bugs/upgrade difficulties. I believe this is on the docket for 2.1 or 3.0 depending on whether it requires a breaking change (that said we plan to release much more frequently going forward so we're planning to have this feature in 2021)

I would say at this point Airflow is leaning pretty heavily on being a data tool. I wouldn't recommend it for something like CI/CD for example. Do you have a use-case in mind?

Thank you Holden! :D

I seriously love that you're an Airflow user since your spark talks first got me into OSS.

Yeah we've been benchmarking for a while, it is VISIBLY noticeable. You're gonna love it :)

If you're switching to python operators you should check out the TaskFlow API. You can basically build python operators with just python functions and decorators.

Hi y'all! Airflow PMC here!

Feel free to AMA about Airflow's new features/the roadmap going forward!

Hi y'all! Airflow PMC here!

Feel free to AMA about Airflow's new features/the roadmap going forward!

I would recommend you check out Airflow 2.0. It's a pretty major rebuild in a whole lot of ways (new UI, new DAG API, up to TEN TIMES faster task execution, multiple schedulers at once). I've actually had friends prepared to pick Prefect over Airflow until they tried 2.0. We put a lot of work into it, including extensive QA time to ensure that it runs reliably.

Disclosure: I'm on the Airflow PMC.