Re:Show HN: Workflow orchestrator in Golang
by harshadmanglani on Monday March 04, 2024 @10:47PM
(#39599199)
Thanks for the inputs! I'll add some examples in the documentation.
- with control on the database reader yourself, i think you should be able to find a way around saturation/desaturation?
- again, the fairly certain you can limit DB readers in the DataStore interface you pass to the orchestrator. I'll think about the cpu and memory share and if there's a way to expose that.
A major concern I've always had with workflow orchestrators is the versioning of workflows. Think about long running workflows (>2 days). If you change your workflow logic, what happens to the existing ones that haven't completed yet?
Everyone handles this differently, and I've been thinking about a generic way to do this. Thoughts?
Would also love to see some examples.
Concerns I’ve had to deal with in the past (if these are solved, examples would go a long way):
- max queue size between elements is important. Don’t want the database reader role to outrun the workers, but also never want them desaturated
- some dynamic limits for local to a role (limit DB readers to limit connection pool) and global (how much cpu and memory share does a job need before starting)