Re:Airflow and the Future of Data Engineering: A Q&A
by somewhatoff on Wednesday March 01, 2017 @04:47PM
(#13768761)
Thanks, that's really interesting. The usage of 'pipeline' to describe both sorts of system made me think there was a lot of overlap, but I'm understanding now how they are complementary.
We operate at a higher level: orchestration. If we were to start using Apache Beam at Airbnb (and we very well may soon!), we' use Airflow to schedule and trigger batch beam jobs alongside the rest of our other jobs.