Ask HN: How do you handle long-running workflows at your company?

Re:Ask HN: How do you handle long-running workflows at your ... by devedlee on Monday May 14, 2018 @12:34PM (#17066874)

We developed and use Argo (https://github.com/argoproj/argo), a Kubernetes-native workflow engine. Argo is currently used by companies like Cyrus Biotechnology, Gladly, Google, Intuit, and NVIDIA. Currently collecting use cases and requirements on a Kubernetes-native eventing framework for Argo (https://github.com/argoproj/argo-events/issues/1) to make it easier to kick off workflows.

[ Parent | on HN ]

Re:Ask HN: How do you handle long-running workflows at your ... by stpedgwdgfhgdd on Monday May 14, 2018 @02:40PM (#17068217)

Does Argo support recovery? In the sense that if a workflow step or the workflow engine crashes halfway, the last (idempotent) action is retried?

[ Parent | on HN ]

Re:Ask HN: How do you handle long-running workflows at your ... by jessesuen on Monday May 14, 2018 @05:27PM (#17069605)

I work on argo. The workflow-controller is very tolerant to crashes and designed to be this way. Workflow state is captured in the workflow CRD object (in k8s etcd). Because step names are formulated, in the event of a crash (say before the created pod is persisted in etcd), when the controller restarts and tries to schedule the pod again, it hits an AlreadyExists error and understands how to handle this. Thus, workflows are idempotent in crash scenarios.

[ Parent | on HN ]

Re:Ask HN: How do you handle long-running workflows at your ... by devedlee on Monday May 14, 2018 @04:49PM (#17069324)

The latest Argo release (https://github.com/argoproj/argo/releases) supports resubmitting workflows with "memoized" steps.

[ Parent | on HN ]