Why Durable Execution Should Be Lightweight

Re:Why Durable Execution Should Be Lightweight by martinpeck on Monday February 03, 2025 @04:14AM (#42916389)

I think the example given in this blog post might need a "health warning" that steps should, generally, be doing more than just printing "hello".

I can imagine that the reads and writes to Postgres for a large number of workflows, each with a large number of small steps called in a tight loop, would cause some significant performance problems.

The examples given on their main site are a little more meaningful.

[ Parent | on HN ]

Re:Why Durable Execution Should Be Lightweight by KraftyOne on Monday February 03, 2025 @09:54AM (#42918747)

Yes, that's totally fair. Usually, a step is a meaningful unit of work, such as a API call that performs an external state modification. Because each step is a fair chunk of work, and the overhead is just one write per step, this scales well in practice--as well as Postgres scales, up to 10K+ operations/second.

[ Parent | on HN ]

Re:Why Durable Execution Should Be Lightweight by CMCDragonkai on Monday February 03, 2025 @11:47AM (#42920121)

I feel like you can generalise this to any transactional key value system, which can scale better.

[ Parent | on HN ]

Re:Why Durable Execution Should Be Lightweight by dalyons on Monday February 03, 2025 @03:21PM (#42922489)

Your intuition is good - having worked with something similar to this, it works great but does not scale very well. The step journaling is pretty brutal to postgres/rdbms , and you hit vertical scaling limits quicker than you would like

[ Parent | on HN ]