Show HN: A distributed workflow engine written in Go

Posted by kindaworkz on Sunday October 01, 2023 @03:29PM
from the show-hn dept.

I'd like to share a project I've been working on for the past few months.

It's a distributed workflow engine written entirely in Go.

Some highlights:

* Tasks are executed in a Docker container * Can run stand-alone or distributed * Highly extensible. * Able to enforce limits (CPU/RAM) per task. * Web UI

Would love the get your feedback on it, and find out if this could be useful.

( 3 comments | on HN )

Re:Show HN: A distributed workflow engine written in Go by importsaas on Monday October 02, 2023 @07:20AM (#37736744)

Potential frenemy here :)

First up, congratulations on the launch, and project. The timing and need for such a project is certainly apt.

Sharing my feedback based on spending 30 minutes as per the guide.

- a config with an example could be introduced earlier. Currently it is deeper in the docs.

- docker compose example could be introduced earlier. Currently it's just in the repo.

- would be great if the destination of the job could also be dynamic. Currently you have a nice yaml way to run SQL/go/bash. I think a job is truly potent when it can pull data from multiple places, but also push that to multiple places as well.

- it's great that it is thinking distributed from the get go.

- having a queue support was smart. having S3 has a source/destination would also be interesting

- how would you do safely do multi-tenant? Eg a SaaS decides to use tork, having 10 customers. I want to run a similiar job for each, and in a way that data doesn't get mixed up.

- hit some errors using an old docker engine. Something I haven't run into for many other projects that can run a docker compose version:3

    ERR node ... failed health check error="Error response from daemon: client version 1.42 is too new. Maximum supported API version is 1.41"
    WRN error getting CPU usage error="not implemented yet"
    DBG received heartbeat cpu-percent=0 heartbeat-time=1696246437 hostname=temp.local node-id=... status=DOWN

- Am wondering if this could someday be a good alternative to swarm/portainer to run long living containers. That with the workflow may help in devops

- how do you schedule jobs. Wasn't clear after seeing examples in 30m

- I notice your git repo deletes branches every few days. So early in a project maybe it can be avoided?

- web ui is promising. Reminds me of inngest territory. Didn't have time to see what the other two tabs. Maybe add those screenshots.

- it's not clear what you mean by extensible. Because of being able to run any image, bash?

- docs could have a section for CRUD like GET requests to see running jobs. Filter by job status, filter by date.

Godspeed!

[ Parent | on HN ]

Re:Show HN: A distributed workflow engine written in Go by jekude on Tuesday October 03, 2023 @11:15AM (#37753015)

Would I be able to use this to run untrusted code from my users? I have had an idea for a game I'd like to eventually build where users write code to control their character. If I send the code to my server running Tork, what are the security risks?

[ Parent | on HN ]

Re:Show HN: A distributed workflow engine written in Go by kindaworkz on Tuesday October 03, 2023 @08:49PM (#37759581)

I actually write a whole post about that exact use case using Tork: https://dev.to/acoh3n/lets-build-a-code-execution-engine-4kg...

Let me know if this helps.

[ Parent | on HN ]