News for hackers. Stuff that matters.

A comment on Cadence: Uber's Workflow Orchestration Engine.

Re:Cadence: Uber's Workflow Orchestration Engine by iblaine on Tuesday April 23, 2019 @06:16PM (#19733529)
Workflow engine = can support hundreds of parallel workflows. This includes airflow, luigi, dagster, appworx, are used to manage data and are typically processes that run in minutes to hours. Orchestration engine = can support millions of parallel workflows. This includes Uber Cadence & Netflix Conductor, are used to manage services and are typically processes that run in microseconds to minutes.
[ Parent | on HN ]
Re:Cadence: Uber's Workflow Orchestration Engine by mfateev on Tuesday April 23, 2019 @06:25PM (#19733578)
Cadence does support processes that run unlimited time. We have workflows in production that are always running. For example there are services at Uber that have an always open Cadence workflow per rider.
[ Parent | on HN ]
Re:Cadence: Uber's Workflow Orchestration Engine by viswabharathi on Friday April 26, 2019 @12:25PM (#19759188)
Uber could've used Cadence in place of Piper/Airflow isn't it. Can you shed more light on where both of these fit.
[ Parent | on HN ]
Re:Cadence: Uber's Workflow Orchestration Engine by iblaine on Tuesday April 23, 2019 @06:35PM (#19733674)
Very interesting. Can you give an example where you'd want to track a process that takes days or months to execute?
[ Parent | on HN ]
Re:Cadence: Uber's Workflow Orchestration Engine by mfateev on Tuesday April 23, 2019 @06:40PM (#19733712)
For example the Uber loyalty program needs to accumulate points (similar to airline points). A customer workflow receives trip completion events and updates the state accordingly. When a certain number of points is reached some actions (mostly calls to downstream services) are executed.
[ Parent | on HN ]
Re:Cadence: Uber's Workflow Orchestration Engine by chrischen on Tuesday April 23, 2019 @10:44PM (#19735119)
For this use case, could it be solved with an asynchronous task system such as Celery? What advantages does Cadence offer for something like this, which seems to be repurposing the scheduling system of Cadence to process instantaneous events?

Also how does it listen for events? From polling?

[ Parent | on HN ]
Re:Cadence: Uber's Workflow Orchestration Engine by mfateev on Tuesday April 23, 2019 @11:06PM (#19735208)
It could be solved by Celery, but it would also require a database and Celery doesn't scale that well unless runs on top of Redis which is not really fault tolerant. Also actions invocation with guarantees and exponential retries is not trivial with Celery.

Actually at Uber large number of services are being migrated from Python/Celery to Go/Cadence.

[ Parent | on HN ]