News for hackers. Stuff that matters.
Re:Lessons learned from running Apache Airflow at scale by idomi on Monday May 23, 2022 @01:45PM (#31482350)
When it comes to scale and DS work I'd use the ploomber open-source (https://github.com/ploomber/ploomber). It allows an easy transition between dev and production, incrementally building the DAG so you avoid expensive compute time and costs. It's easier to maintain and integrates seamlessly with Airflow, generating the DAGs for you.

There are no comments on this comment yet.