News for hackers. Stuff that matters.
Re:Introduction to Simple Workflow Service (SWF) by tiagod on Wednesday July 15, 2020 @10:20AM (#23845546)
I've been using temporal.io, which afaik is developed by some of the people who built SWF (and Cadence at Uber), and I think it's great. I'll probably use it for a ton of stuff in the future.
Re:Introduction to Simple Workflow Service (SWF) by sagikazarmark on Wednesday July 15, 2020 @11:37AM (#23846764)
We (at Banzai Cloud) use Cadence for managing infrastructure. Workflow as code is a huge thing for us, because we can use the official SDKs for cloud providers, but still have the flexibility of implementing our own logic easily and in a testable way.

Temporal is even better as it replaces Thrift with gRPC and adds tons of improvements.

Re:Introduction to Simple Workflow Service (SWF) by arrayjumper on Wednesday July 15, 2020 @10:45AM (#23845888)
We have a bunch of services which orchestrate long running workflows so temporal is quite interesting to me. I've been lurking in the temporal slack for a while (and before that in the cadence slack).

Would be interested to know what kinds of things you're using it for/what scale/which language client?

Re:Introduction to Simple Workflow Service (SWF) by ttsda on Wednesday July 15, 2020 @11:03AM (#23846177)
I work for an accounting company with a few hundred workers, who every month had to submit a lot of files, and collect a lot of documents from human-only websites.

About two years ago I built a custom workflow engine in python where I had a few primitives (filesystem, database access, browser (puppeteer scripts), etc), and would define a workflow using those primitives using a YAML DSL.

In a couple weeks I ported the whole thing to Temporal (Java client), where each of those primitive groups is a different activity, which runs in a separate worker. I have a browser worker, which runs in the only servers that have internet access, a filesystem worker, which is on a Windows machine serving as an entrypoint to the company's windows filesystem, a database worker, etc.

This is all very scalable, and I can run as many of each worker and the engine itself as I like.

Currently the scale is not too big though. I launch a few "main" workflows in certain days of the month that will then spawn up to 4 thousand child workflows which run simultaneously with no problem.

Re:Introduction to Simple Workflow Service (SWF) by teraku on Wednesday July 15, 2020 @10:30AM (#23845682)
This looks awesome, but I don't understand what this has that airflow, prefect or luigi don't have?
Re:Introduction to Simple Workflow Service (SWF) by bbri06 on Wednesday July 15, 2020 @12:30PM (#23847588)
Scale, interactivity, and durability are certainly part of this discussion. You can use Cadence/Temporal for purposes you just really couldn't use Airflow for. For instance, you could implement this entire actor-based auction example from Akka with Cadence/Temporal https://doc.akka.io/docs/akka-enhancements/current/persisten... There is absolutely no way you could do that with Airflow.

To understand why Cadence is so useful, you have to go a bit beyond the traditional ETL scheduling usages IMO.

Re:Introduction to Simple Workflow Service (SWF) by ttsda on Wednesday July 15, 2020 @11:04AM (#23846198)
In SWF/Durable Functions/Cadence/Temporal, The entire workflow is defined as (deterministic) code, which is orchestrated by the engine.
Re:Introduction to Simple Workflow Service (SWF) by teraku on Wednesday July 15, 2020 @11:16AM (#23846379)
As as opposed to... having undeterministic code? Workflow as code is also the theme of airflow, luigi and prefect.
Re:Introduction to Simple Workflow Service (SWF) by firdaus on Wednesday July 15, 2020 @11:25AM (#23846587)
With Temporal/Cadence you write (mostly) plain code to implement your business logic rather than using code to define DAGs.

This discussion has more on Airflow vs Cadence: https://news.ycombinator.com/item?id=19732447