In concrete terms if your app has functions that do very little work but run for hours or days at a time blocking on external resources, and it is acceptable to add 100+ms latency + a serialization boundary to every function call, it is acceptable to take on a bunch of other tooling complexity, and you are willing to sacrifice verifiability of the code, then it might be a fit. In those situations the structure of it (or DBOS or similar) can help standardize that part of your app. Unlike a regular state machine based approach it's easier to represent branching logic and so forth in imperative code. I think its popularity is like the Mongo NoSQL fad, it looks convenient up front but in practice is very painful in most situations. Having used it a fair amount I would recommend starting with actors and state machines, especially in this age of robot code where it's easier to apply heavier approaches to testing in verification.

It’s an async job runner that scales well. Yeah they advertise a lot of fancy features, but really what they are selling is solid failure-resistant infra. Companies could set the same thing up with an in-house team, but it’s pretty great to be able to click a button and start running workloads at tens or hundreds of thousands of RPS without needing to figure out the internals.

Can I understand it as a performant celery with a UI?

Not the same thing, because celery and the like are sdk-level abstractions that need you to plug in your own queue (usually Redis, Kafka) and all other components. The Temporal server deployment includes the full control plane, queue, persistent database, UI and everything else. So you only need to manage your worker boxes.

A Rust equivalent to Celery would be something like Apalis [0]. Where every job is just an async function, similar to a Celery task.

[0]: https://github.com/apalis-dev/apalis

With better ergonomics around handling failures.

Durable execution comes with overhead that you'll need to determine whether or not fits your use case. If you're looking for raw speed then durable workflows are going to be a steep trade off. Most workflows should be durable by default, though, so it is more the rule than the exception.

I think it's useful for implementing Sagas https://temporal.io/blog/saga-pattern-made-easy

Basically, a multi-step event handling system, where the data could be spread across multiple databases/systems, so there's no way to rollback a transaction atomically across all the databases. Instead, you explicitly codify "compensations" to undo your previous commits so that you eventually end up in a consistent state.

Temporal is the orchestrator/framework/library to implement the above in an easier way.

Does it actually work?

In a past job, they tried to implement a similar thing on much lower scale with bidirectional database migrations.

Fortunately, they were mostly ran in one direction.

IME, it works, but at the cost of a pretty large amount of overhead, both in performance and (I think more importantly) cognitive.

Honestly I never did come to a satisfying conclusion on whether I thought it was worth it. The teams I worked on that used it found it neither simple nor easy to use, and I was never sure whether we were really reaping the benefits of correctness in the face of failure at the scale we were running. We eventually migrated everything off of it, and everyone was happier, but perhaps (probably) we had more lurking bugs in distributed failure cases. But to my knowledge we never tracked down an incident to the kind of problem that temporal solves.

I guess I'd say that I like durable execution (or at least temporal specifically, it's the only system like this that I've used) in theory, but not really in practice.

The fintech (now bank) that I worked in couple of years ago utilized this pattern pretty effectively.

The overhead of doing something in this way is quite large, but, as this handled literal money, it was definitely worth it.

Yeah, I’ve seen it working. It does work well to implement a multi-system transactional workflow broken down into retryable steps. So, eg, if you are operating a SaaS and need a workflow for new customer signup to wait for multiple steps that might last for hours or days like: validate credit card info, wait for email confirmation, add entries to customer database, set up workspaces in downstream systems, spin up cloud resources, initialize systems, notify customer, start billing cycle, etc. and if any step fails, you might want to retry or send alarms or internal notifications, or roll back other steps, or kick off other workflows. Then you can define all of that, in code. Each step can be different runtimes run on different worker systems, etc, all coordinated through a central database.

You can scale the workflows up and down to as complex or simple as you want for whatever your business is. The example above is one I’ve seen, but it works on a smaller scale as well. But you have to write the implementation code in very specific ways to get the benefits, and the overhead ends up being very difficult to plan and reason about, and a lot of business logic actually gets hidden amongst the weeds of the tool’s overhead.

When you write traditional distributed systems/cloud native code you have to handle all the failures and retries somewhere. Anyone who's run an Ansible playbook or similar and had a failure halfway through leave the state of the systems in a weird half-state is familiar. Or dealt with spot preemptions or nodes failing or getting unlucky with OOM killer or hardware failures or a myriad of other failure modes.

In Temporal, you use their SDK to mark which code is either:

- Deterministic without external dependencies on network, disk, clock, etc.

- Non-deterministic (e.g. accesses a filesystem, dependent on clock time, talks over the network)

You can write the code without handling flakiness or retries and the Temporal control plane handles all the progress tracking and retries for you. Progress is tracked at the individual line of code for deterministic code, or for non-deterministic code, tracked at function boundaries defined by the programmer. You buy into more complexity upfront, but it makes the application code way simpler and easier to manage overall.

They do a lot of other cool stuff on top of all this, and their Temporal Worker Controller architecture is particularly well suited to running massive scale processing/AI workloads on Kubernetes (handles a lot of stuff like autoscaling without interrupting work, rainbow version rollout, etc.)

Say you've got a process that spans a bunch of services and touches the real world. Some ecommerce check out for example.

You need to reserve some inventory, charge a card, eventually email a customer and steps can go wrong. So you have queues, and retries and ways to back things out, undo changes.

To my understanding, Temporal's idea is to factor that part out, the queues and retries, and offsetting actions, so you write the logic and not the workflow orchestration.

Asking what temporal is? The founders (Maxim Fateev and Samar Abbas) originally made Amazon Simple Workflow Service (SWF), Azure Durable Task Framework (DTF), and Uber Cadence. Temporal is the latest in the long line of workflow systems they've built. It's a task engine with state, recovery and highly scalable. It's cool stuff, and it simplifies many business workflows when it comes to recovery and scaling. There are many similar systems on the market today that mimic durable functions but Temporal is arguably the best.

Thank you for all the explanations. I think I have better understanding now. Another question, at what scale (in terms of number of services, and maybe throughput? what other dimensions?) should we start considering using temporal?

I think scale is less important than other considerations. For me the biggest motivator is when the use case prefers doing something exactly-once, the downside risk of failure is high, and robust failure handling is moderately complex. The canonical example I use is healthcare communications; others use payment/fulfillment.

Also having a unified control plane is handy even at a pretty small scale if your alternative is going full “cloud native” on hyperscaler microservices. The ability to see what’s happening when and where across a single workflow run is a dream compared to all the traditional logging approaches I’ve seen.

"Why is it such a big deal?"

It's used by OpenAI.

temporal makes you split up your async tasks explicitly into a message (a request to fulfill a task, thrown up to a temporal server), and a task handler (pulling a tasks-request from a queue, from the temporal server), rather than both wrapped in a vanilla request/response cycle.

temporal centralizes all the error handling, retry handling, ... some web pages to manage failed, pending tasks.

it's been good for us, but a real step function in complexity of the app.

This is a rough explanation, but generally it does a couple of things that are helpful.

Functionally, it's a "workflow" runner (ex - you can mostly treat it like a queue, where you've got workers that are picking up work to do).

But it wraps a couple of pretty handy features on top like:

- It preserves most arguments to actions, and it makes system details deterministic for retries (ex you can re-run a workflow at a later time, and temporal will make sure the code sees details like date and time as though it were the original run, and will yell at you if you try to write code that won't be deterministic on retries)

- It supports very long waits easily. (ex - very easy to have a workflow do a couple things, wait a week, then do some more things).

- It has decent profiling and UI tools

- It can "restart" a failed workflow deterministically from the step at which it failed (using details from the original point)

---

Basically - it's a background worker service that's put a lot of time and thought into ways to handle failures better.

It absolutely still has some considerable pain points though, and I find it difficult to use for larger tasks (ex - their message gRPC size limit of 4mb is a b*&^% to work around, since it often breaks a lot of the utility they provide, and the history cap at 50mb is also really painful in certain situations.)

Really - I think it was just the right tool at the right time to make calling LLMs with long waits relatively easy and somewhat foolproof.

Another cool thing is it handles version rollouts. Say a customer started some kind of business process on v1.1 of your code, and then you deploy v1.2. You can configure whether that customer should continue that process on v1.1 - even if it's something that takes a long time in the real world - or whether they should be upgraded to 1.2. And it's not just one version but you can do this with an entire rainbow of versions across your business (think A/B testing, custom workflows for different use cases, lots of dev and staging environments, etc.)

> So... like exception handling?

Yes...with eventual consistency.

If your process OOMs, if the machine restarts, etc.

Plus some other distributed execution advantages.