Re:Lessons learned from running Apache Airflow at scale
by commandlinefan on Monday May 23, 2022 @12:10PM
(#31480866)
I tried installing Airflow locally to just play around with it and make sense of what it's good for and finally gave up after a few days - the install alone is insanely complicated, with lots of tricky hidden dependencies.
Re:Lessons learned from running Apache Airflow at scale
by always_left on Monday May 23, 2022 @12:18PM
(#31481008)
Did you try installing with docker? You would just download docker, `docker-compose up --build` and you'll be good to go locally (usually)
Re:Lessons learned from running Apache Airflow at scale
by idleprocess on Monday May 23, 2022 @01:00PM
(#31481648)
I can second this. We were up-and-running with Docker on our dev machines in just a few minutes. A native installation involves substantially more setup (Python, databases, Redis and/or Rabbit, etc.). The published docker-compose file will handle all of that for you. We have a very small data engineering team and have been able to move very quickly with Docker and AWS ECS (for orchestrating containers in test and prod environments).
Re:Lessons learned from running Apache Airflow at scale
by flowair on Wednesday May 25, 2022 @10:34AM
(#31505143)
download the astronomer CLI and it will spin up a local dockerized instance of airflow in about 1 minute tops
https://docs.astronomer.io/astro/install-cli
I am part of one such team. We were using Windows Task Scheduler on a windows VM to run jobs, we figured it would be a nice idea to (dramatically) modernise and move to airflow, but we grossly underestimated the complexity, learning curve, and surrounding tools it requires. In the end we (data science team) didn't get a single production task up and running. The data engineers had much more success with it thought, probably because they dedicated much more time to it.
Will look forward to trying AWS Step Functions.