News for hackers. Stuff that matters.
Re:Effective Airflow Development by james_woods on Wednesday August 19, 2020 @06:17AM (#24208860)
I am using Apache Airflow since a couple of years now and the biggest improvement was the addition of the Kubernetes Operator. You basically keep a vanilla Airflow installation and the custom code is encapsulated and tested in containers. This simplifies it a lot.
Re:Effective Airflow Development by recov on Wednesday August 19, 2020 @03:38PM (#24215215)
This is what we do - it's great for decoupling the two. Any heavy work is run in GKE with some custom operators. It also makes on boarding non engineers much easier as they don't have to worry about connections/credentials/etc.