Then I tried their Cloud offering and was appalled at their pricing. I burned through the $1,000 free credits before I even got something to production. Didn't want to bother with running a local Temporal, either.
Best solution is to just take inspiration from their architecture and then do it yourself in Postgres, IMO.
https://www.enterpriseintegrationpatterns.com/patterns/messa...
Very happy we made the switch.
Temporal is, in my opinion having run it in prod for over a year - poorly designed, slow and ridicliously heavy infra wise.
If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck.
Try running their own benchmarks, the numbers are pathetic.
Their sales team is also absolutely appalling and desperate.
From a Developer standpoint, the SDK is quite nice though.
Don't get trapped into nexus, and if the sales team call you make sure legal is in the room.
https://github.com/temporalio/temporal/blob/e22e6304b3c4a409...
https://github.com/temporalio/temporal/blob/e22e6304b3c4a409...
Temporal does a crazy amount of database operations and all of these are behind that mutex.
Oh, and you can't change the shard count on existing clusters.
Great stuff.
Where are the “millions” on infra going? It’s a handful of services and a Postgres?
> Their sales team is also absolutely appalling and desperate.
You said “on-prem”. It’s open source; why are you dealing with their sales team?
> If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day…
If “millions” were required to obtain such tiny scale, I’d agree there’d be a massive problem. No one would use Temporal; it would be a complete waste of resource. If this were true.
Postgres doesn't scale at all four our workload, so you're into cassandra.
For a medium sized deployment, you're looking at 200+ vcpus, and then lets say standard dev/uat/prod. So now you're at 600 cpus. Now you need two geographic regions, dev can stay in one place, so now you're at 800. Want a failover cluster for prod? Have another 200 cpus.
and 200 CPUs is a medium deployment, assuming something like 36 cpus per cassandra node, then say 4-8 per instance of matching, worker, history, frontend. Then all your other components around it, ingress controller, service mesh, etc.
There's a million a year easy, for a small deployment.
Our prod one is 4x this size.
We need a 12 node cassandra cluster for this, with 64cpu nodes. So no, it's not a couple of services and a postgres.
Sales team, as we are an enterprise, and they want to extract money from us.
If you have 200 WF’s/sec each with 200 events, it sounds to me that you have a sizeable amount of work flowing through this system. 17 million workflows per day? Can I call these transactions?
Do these transactions add value to your business? Do you need durable execution for all these workloads?
Temporal is just a tool; and like any tool it can be misused. For the classic “book a hotel + airline, handle the partial failures” case, 17 million bookings a day would imply you should be thrilled with Temporal.
If you are using it to perform WAF in a firewall; you would be less thrilled. The scale you are describing, and that you aren’t super excited about the incredible amount of money pouring in, makes me question if the use-cases are fitting the tool.
Ballparking: 200 events/workflow, 200 workflows/per day and assuming 1 event = 1 cloud action[1], that is 1.2M or so actions per month. The $100/month plan includes 1M actions each month, and even the pay-as-you pricing when you exceed that is $50 per 1M actions[2].
Temporal Cloud seems extremely cheap for your use case, even if I'm off by a factor of 10. Is there a catch? You still need infra to run your Temporal workers, and I assume there are storage and other costs, but I assume action usage is the majority of it.
1. Not sure exactly what constitutes an "Action". At a glance, seems like most events have a corresponding action(?) and a subset of those actions are actually billable(?)
2. https://docs.temporal.io/cloud/pricing#payg-action-pricing
Temporal was a bad fit for us, and we regret it deeply.
Given the open source nature of the product, I'm not sure what vendor lock-in you mean. Migrating off cloud to your own infra again is doable. https://daylight.ai/blog/how-we-migrated-off-temporal-cloud-...
As our CEO and co-founder Samar Abbas stated in the keynote at this year's Replay conference (https://youtu.be/BxEB7Y2U9oU?t=532), Temporal Cloud is currently handling 35 billion workflows per day. If you assume that each Workflow averages 10 actions, that translates to more than 10 trillion actions per month.
Later in that keynote, a VP from OpenAI talks about the scale at which they are using Temporal Cloud. That'll give you a sense of the volume for a single customer (obviously a very large one).
What's the deal? It couldn't harm just listening to sales, could it?
I presume legal would it be involved before anything is signed in any case?
I really like running a thin rest API in front of it inside your vpc or k8s cluster or whatever to help with event driven triggers so that they don't have to worry about Temporal auth and checking workflow status if there is any decision making around that. This helps keep your event as logic-free as possible.
Let me give a vague example: you have some sort of db trigger, and this trigger either acts directly or puts the event on a queue, your handler calls the thin rest api with the necessary event details, rest API can make the decision if this starts a workflow, signals an existing one, or ignores it (the pattern for this can vary based on the situation, but SignalWithStart is common for me or just dropping if the event is not worthy of starting a workflow and no workflow for that <ItemYouCareAbout> exists).
Then the parent/child workflow ability is very valuable when you need to orchestrate different self-contained behaviors for a single object's lifecycle, with cancellability when an external factor changes the trajectory of an object.
Long, vague story short, I find it very powerful and easy to work with and has really helped move lifecycle logic out of APIs where things can easily become riddled with debt and precarious to manage. I agree with you that it helps follow more best-practices instead of just throwing logic some place that seems easy but becomes a hidden trap later.
Main tradeoff is lower performance. Or at least, you’re going to be limited to what you can push through Postgres. If that’s sufficient for your needs DBOS is great.
Tens of thousands of workflows per second
I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then download it in the workflow.
What is your experience with DBOS? How does it compare to Temporal in terms of operational complexity, feature parity and anything else