The idea behind airflow is great. What sucks is people using it to do heavy processing. Maybe with serverless/k8s airflow could fan out the processing to a cluster to allow for flexibility. But then, I guess you end up re-writing spark et-al.