Lessons learned from running Apache Airflow at scale

Re:Lessons learned from running Apache Airflow at scale by commandlinefan on Monday May 23, 2022 @12:10PM (#31480866)

I tried installing Airflow locally to just play around with it and make sense of what it's good for and finally gave up after a few days - the install alone is insanely complicated, with lots of tricky hidden dependencies.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by always_left on Monday May 23, 2022 @12:18PM (#31481008)

Did you try installing with docker? You would just download docker, `docker-compose up --build` and you'll be good to go locally (usually)

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by idleprocess on Monday May 23, 2022 @01:00PM (#31481648)

I can second this. We were up-and-running with Docker on our dev machines in just a few minutes. A native installation involves substantially more setup (Python, databases, Redis and/or Rabbit, etc.). The published docker-compose file will handle all of that for you. We have a very small data engineering team and have been able to move very quickly with Docker and AWS ECS (for orchestrating containers in test and prod environments).

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by flowair on Wednesday May 25, 2022 @10:34AM (#31505143)

download the astronomer CLI and it will spin up a local dockerized instance of airflow in about 1 minute tops https://docs.astronomer.io/astro/install-cli

[ Parent | on HN ]