The most accurate answer would be from comparing various ESB products to Workflow Orchestration products, as every vendor/product will have slightly different opinions on where their responsibilities are.
No, ESB is the messaging plane, workflow orchestration is a higher level service, on top of the messaging plane.
I'd like to have some easy to set up orchestration/job scheduling engine in my team so we can help clean up the tangled mess that we are in but something like Ansible seems like too much work to setup and add more jobs over time. I tried Activebatch and I wish there was some free or cheap alternative of it.
What are the types of problems you're hoping to untangle, more specifically? I assume you want something like... A workflow where each step maps to some provisioning process?
If that guess is reasonably true, then I immediately think of Terraform. You can specify dependencies and write hooks that trigger callbacks when certain steps are called. We use Terraform a bunch under-the-hood for Refinery, and it's great. I haven't used Ansible (only read about it) so I can't contrast them very well.
If you want to chat some more, I'd be curious to hear more about what you're trying to build and see if there is a way we could collaborate. Or if I can offer any recommendations for software that I know of. My email is free at refinery.io :)
In a nutshell it's just a runner and uses a project called moleculer which is a microservice framework that I'm more or less just using as an rpc client to execute the tasks of the workflow.
One thing I've been debating with myself is how I should work with the dataflow. Would you say it's better to have the results of every node merged into a singular context across an entire flow that is passed to every other node/block (ie. always one input, the context, one output the context), or would it be better to explicitly declare and pass in inputs/outputs.
Here's one project that does it that way: https://github.com/danielduarte/flowed
One benefit of this seems to be that nodes can run in parallel the moment their dependencies are met without having to care about each other.
I poked around on your site and I like what you have to offer. You appear to do it the singular context way which I suppose makes sense because it seems like every block is an individual lambda. This also seems like overkill for my purposes because I'm interested in being able to do granular things if I wish such as a simpe rule action. I'm not sure having an individual lambda to run "if email exists return true" would be practical. That and the warm up time/latency.
The other thing is managing something like npm dependencies might be annoying across blocks.
Being able to create arbitrary API endpoints is nice though.
I wish there were something a little more in between than all of these enterprisey offerings.
Argo seems more CD focused right now whereas Tekton really is a “toolbox”/make what you want.
"Tekton is a powerful and flexible open-source framework for creating CI/CD systems, allowing developers to build, test, and deploy across cloud providers and on-premise systems."
Do you know why they are switching, out of curiosity?
The market is definitely very young and I don't think anybody has really nailed down every use case.
There are the parts of the market targeting "business model" use cases like Camunda/Zeebe.
Then there are ETL-style systems like Airflow for dealing with massive data processing.
And still you've got the CI/CD side of things like Argo/Tekton for automating complex build systems/running tests.
Then for systems like Netflix Conductor, Uber Cadence, and AWS Step Functions (among others), you have systems trying to abstract on top of existing complex systems (microservices, etc).
That's not even including low-code spaces like Zapier or IFTTT that try to target making integrations trivial.
It's a crazy world!
At this point in my experience for both (I have much, much more time with Tekton though), Argo is good for CD. It's visualization and syncing of environment charts (we use Helm) is useful.
Tekton is amazing for defining highly complex build/deploy workflows (which we have). Along with Tekton's sibling project Triggers, you can have full end-to-end PR -> Build -> Deploy -> Feedback into Slack/Monitoring/Alerting/etc.
If I had to make a comparison (with my current knowledge level), I'd say Argo is like Angular, batteries loaded with addons you can include if you wish. Tekton is like React, you get pieces and build them into whatever you want.
EDIT: nevermind, i see your getting started docs have lots of video clips.
Glad you found the docs here[0] (for anybody else who is curious).
We're still iterating on Refinery a lot and trying to find product market fit. If it doesn't make sense or you're confused, that feedback is super helpful for me. The goal is to build something people actually want to use and it's an iterative process to get there!
We recently added support for Git-based development to address testing. The idea there is that each block is just a chunk of code living in a Git repo, so you can use whatever testing tools you'd like to.
It's actually pretty slick -- it's bidirectional. You can use Git to make a commit, like normal, and then refresh the editor to see it. And you can make also changes in the editor and it will commit + push to Git. If you want to play with that, I can share details on how to enable it (it's fairly hidden in the UI atm).
In terms of testing workflows together as a chain, that's still TBD. It is something that we'd like to ship eventually though. Along with staging environments and canary deployments.
On our immediate roadmap is to get the core of the tool open-sourced so that people can start playing with it outside of our hosted platform. We have it gated behind a credit-card form right now to fight fraud and because of some technical limitations.
Conductor is a cool piece of tech, and it's a well-established player in a rapidly growing space for workflow engines.
I used to work at Uber and that company had microservice-hell for a while. They built the project Cadence[0] to alleviate that. It is similar to Conductor in many ways.
One project to watch out for is Argo[1] which is a CNCF-backed project.
There are also some attempts[2] to standardize the workflow spec.
Serverless adds a whole new can of worms to what orchestration engines have to manage, and I'm very curious to see how things evolve in the future. Kubernetes adds a whole dimension of complexity to the problem space, as well.
If anybody is interested in chatting about microservice hell or complex state machines for business logic, I'd be excited to chat. I'm always looking for more real world problems to help solve (as an early stage startup founder) and more exposure to what others are struggling with is helpful!
0: https://github.com/uber/cadence
1: https://argoproj.github.io/argo/
2: https://serverlessworkflow.github.io/