Re:Airflow 2.0
by timwis on Thursday December 17, 2020 @04:47PM
(#25460931)
Does anyone recommend a simpler alternative to airflow for simple multi step workflows that isolate task dependencies?
[ Parent
| on HN ]
I recommend checking out Conducto. It handles dependencies well by leveraging Docker and it is very easy to get started. It can also handle super complicated pipelines that required scaled up processing by just changing one input on the command line.
Re:Airflow 2.0
by willvarfar on Thursday December 17, 2020 @04:58PM
(#25461061)
One way to interpret the 'simpler' requirement is to suggest Luigi. Its a python framework. Its way simpler if you are a programmer.
Of course a lot of people think of Airflow as 'simpler' than Luigi, because Airflow is all graphical. YMMV.
Re:Airflow 2.0
by morelandjs on Thursday December 17, 2020 @09:00PM
(#25462961)
Really happy with Prefect for smaller workflows
Airflow 2.0 :)