Very cool projects. In general, across all projects, what is the best approach to store/checkpoint the state of the program? I'd imagine something like CRIU but I'd love to hear your thoughts. When do you take snapshots. Do you hook-up to the VM's event loop being drained? How do you store an d replicate these snapshots?

The Temporal/Cadence/SWF operate as libraries any application can include to be able to implement workflow and activity logic. So hooking into low-level VM event loop was not an option. So they rely purely on event sourcing to reexecute the program code from the beginning assuming that the workflow code is deterministic. The library provides various API wrappers to execute multithreaded code deterministically using cooperative multithreading. In the future, WebAssembly can be used as a container as determinism is one of its core features.

What considerations do you put into how to handle versioning? I'm thinking mostly of the "decider" logic.

For example, in SWF + Flow land, I maintain different versions just by keeping the old implementation and using version numbers. For Temporal, your team's recommended way is to have a single decider get the execution's version, and write conditionals into the business logic for both cases (https://docs.temporal.io/docs/java-versioning). Is the change in approach intentional, or just a matter of what's been built so far?

The change in approach is intentional. It is still possible to use the old method, but for the majority of cases, it is just too heavyweight. There are two main problems with versioning entire workflows:

1. Need to keep the workers with old code around. It might be solved for some languages like Java by dynamic code loading, but it is still pretty complex. For long-running workflows dozens of changes can happen during their lifetime, requiring dozens of versions to be present at runtime.

2. When a bug is fixed the old versions do not get the fix as they are inherently immutable. Thus a workflow that runs for a month and has started yesterday is still going to experience a bug that was fixed today.

The approach that Temporal promotes is to version each piece of code independently when needed. This way, there is only one version of the entire code in production to maintain, and bug fixes can be deployed any time and apply to all the open workflows that didn't reach the code that was fixed.

What was the reasoning behind the decision to define workflows as code versus in a configuration format like JSON (think Netflix Conductor) in Cadence and Temporal?

The reason is that workflows are inherently imperative.

Configuration formats work well for domains where declarative programming makes sense. For example, Terraform defines "what should be done" instead of how it should be done. Some workflows can be described using declarative syntax. But these belong to some specific domain. In this context, Terraform HCL can be seen as a workflow definition language for infrastructure deployment.

Temporal/Cadence are targeting nondomain specific workflows, so they have to be imperative. And I strongly believe that JSON/YAML/XML/etc. are awful languages for writing imperative programs. They add no value but introduce immense complexity. And they almost always embed some expression language for condition and other evaluations.

For imperative programming, any general-purpose programming language beats heads down any configuration based language in clarity and other features like IDE support, debugging, unit-testing, mocking. We have software systems built form millions of lines of code, and people still make sense out of them. Just imagine Linux kernel written in JSON instead of C.

Note that some workflow engines define workflows as code as well. Airflow is a notable example of this approach. Note that Airflow uses code to define DAG. But the actual execution engine executes the DAG and not code.

Temporal executes the workflow code in a general-purpose programming language directly without any intermediate conversion to AST/DAG or similar.

Then the question is, what makes Temporal workflow code a workflow? Is any code a workflow? Temporal makes workflow state, including local variables and thread stacks fully fault-tolerant. It is like having a computer with a fully durable RAM. Actually, Temporal provides even stronger reliability as a workflow code is not linked to a single computer is automatically migrated between computers or even different clusters when infra failures (or just new code deployments) happen.

I would recommend checking out the documentation (https://docs.temporal.io/docs/overview) and samples (https://github.com/temporalio/java-samples, https://github.com/temporalio/go-samples) to see it for yourself.

> Temporal/Cadence are targeting nondomain specific workflows, so they have to be imperative.

I'm not sure that follows, although it's interesting! BPMN, for example, is a declarative way of defining processes that isn't domain specific. Did you have a look at that?

BPMN is not declarative. It is imperative. It doesn't say what should be achieved. It gives exact instruction what should be executed and in which order. And a big chunk of those instructions are not really visible in the diagram. For example which parameters each activity takes or shared state updates are hidden.

So it has all the issues that I pointed out above.