News for hackers. Stuff that matters.

A comment on AWS Managed Workflows for Apache Airflow. Parent comment

Re:AWS Managed Workflows for Apache Airflow by seddonm1 on Tuesday November 24, 2020 @05:37PM (#25203829)
I am hoping Argo Workflows (https://argoproj.github.io/projects/argo) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.
[ Parent | on HN ]
Re:AWS Managed Workflows for Apache Airflow by vanpythonista on Tuesday November 24, 2020 @07:28PM (#25204643)
I like Argo and have used both solutions, but I'm not yet convinced that YAML based workflows are superior to Airflow's Python Code as workflows.
[ Parent | on HN ]
Re:AWS Managed Workflows for Apache Airflow by seddonm1 on Tuesday November 24, 2020 @09:16PM (#25205278)
Agree that YAML is not really ideal for this case.

For other readers Argo does support 'Scripts' which means you can implement complex logic similar to Airflow if required but can be a bit clumsy: https://argoproj.github.io/argo/examples/#scripts-results

Additionally, I think we may see other layers on top of Argo in future that may abstract from the YAML similar to how Kubeflow uses Argo for the Kubeflow Pipelines capability: https://www.kubeflow.org/docs/components/pipelines/pipelines...

[ Parent | on HN ]
Re:AWS Managed Workflows for Apache Airflow by kumare3 on Tuesday November 24, 2020 @09:40PM (#25205439)
Have you tried Flyte - flyte.org?
[ Parent | on HN ]
Re:AWS Managed Workflows for Apache Airflow by lxglv on Wednesday November 25, 2020 @04:38AM (#25207455)
Running something on Kubernetes does not make this something highly available. It makes this something randomly restarting.
[ Parent | on HN ]
Re:AWS Managed Workflows for Apache Airflow by seddonm1 on Wednesday November 25, 2020 @06:05PM (#25215287)
Maybe.

In this case Argo uses the Kubernetes distributed state store (etcd or equivalent) and is a stateless service so can deal with failures quite well.

[ Parent | on HN ]
Re:AWS Managed Workflows for Apache Airflow by drewda on Tuesday November 24, 2020 @06:37PM (#25204302)
Agreed. We've had good luck running all sorts of jobs on Argo and wrote up some of our experiences at https://www.interline.io/blog/scaling-openstreetmap-data-wor...
[ Parent | on HN ]