qlong-seattle

Share my recent learnings as an expert of iWF , but a beginner of LLM

Took at quick look at apache nifi. Here is my conclusion but anyone feel free to correct me if I am wrong:

I think they only overlap for a small part but quite different:

* Nifi is focusing on moving/sharing datafile (meaning the size could be big) across the system * Nifi does have some logic control flow to act like workflow, but it's not too closed to be a "workflow system" because it missing a lot of capabilities that a typical workflows should have (see "What they all have in common " in https://medium.com/@qlong/iwf-vs-other-general-purposed-work...)

* iWF is a general purposed workflow engine, could conceptually do data sharing but it is not good at sharing the big amount of data or data files * iWF is more specialized at controlling/orchestrating the business

We don't have any plan to build other SDKs at this moment.

But it's VERY EASY to build an SDK for iWF so I would encourage anyone interested to build one. I will provide all the support.

A SDK of iWF is very lightweight. It only need to invoke the two WorkflowState APIs from hosting a REST/HTTP service, based on the worker API schema. https://github.com/indeedeng/iwf-idl/blob/main/iwf.yaml#L215

There is no much tricky things inside the SDK that you have to do. For client APIs to call iWF server(the rest of the API defined in the Open API schema), just need to build a wrapper on top of the generated code from Swagger (and it can be optional but it will just make it easier to use).

As a data point, it only took me a week to build GoSDK, including all the tests, by just opening the IntelliJ IDE and also Goland IDE to translate the Java code into Golang.

The whole community including myself will really appreciate it.

Yeah this wiki page has exactly what you want to see here:

https://github.com/indeedeng/iwf/wiki/How-to-migrate-from-Ca...

There are lots of things that other workflow engines cannot do:

Reset workflow : important to recover from bad states for any reasons. You may have bugs caused this or because downstream outages. Reset can let you go back like timing machines.

Search attributes: let you search for anything like having unlimited database index.

See more details in this article: https://medium.com/@qlong/iwf-vs-other-general-purposed-work...

You don't have to run elastic.

Cadence/Temporal let you run only with database for majority of the feature/power. ElasticSearch is only for supporting the searching functionality(advanced visibility).

https://docs.temporal.io/visibility

https://cadenceworkflow.io/docs/concepts/search-workflows/

Same as in iWF, you can use a Temporal service without advanced visibility.

E.g. this doc is showing how to run with Temporalite which doesn't have advanced visibility with it.

https://github.com/indeedeng/iwf/blob/main/CONTRIBUTING.md#o...

Again, you still have 90% power and the only thing missing is searching for workflows using "search attributes".

You can still store data in search attributes but they are just not searchable.

And later, you can migrate to a Cadence/Temporal service that has advanced visibility enabled.

I believe so. Welcome to try out. There are only a few concepts you need to learn about iWF and you can start with docker-compose to run a iWF service locally for development.

iWF is not just persisting states, but also able invoking/executing states, searching for attributes(like unlimited database indexes), durable timer, resetting workflows etc.

Without Temporal/Cadence, it's 100+ harder to implement iWF. Even using SWF/Azure durable function is possible to build iWF, Cadence/Temporal is much more powerful than the other two(e.g. no reset workflow, search attributes)

Maybe you can start with looking at the sample for solving the same "subscription" use case:

Temporal:

https://github.com/temporalio/subscription-workflow-project-...

https://github.com/temporalio/subscription-workflow-project-...

iWF:

https://github.com/indeedeng/iwf-java-samples#product-use-ca...

https://github.com/indeedeng/iwf-golang-samples#product-use-...

Note that you need to consider making changes to workflow -- in Cadence/Temporal, you must use versioning like: https://stackoverflow.com/questions/61298415/how-can-i-versi...

And this versioning/determinism constraints is just a small part of the learning curves for complicated project

And I am not the only one pointing this out. Also see the article from the principal of Azure Durable Function team:

https://medium.com/@cgillum/common-pitfalls-with-durable-exe...

There are samples from the README:

https://github.com/indeedeng/iwf-java-samples#product-use-ca...

https://github.com/indeedeng/iwf-golang-samples#product-use-...

Hello, no you are definitely the target users. Anyone who needs to build long running business/application/process is the target user of iWF.

You are right, there were too many context about Cadence/Temporal in the README. I just made a update to move a lot of them to wiki just for the Cadence/Temporal users.You can check https://medium.com/@qlong/iwf-vs-other-general-purposed-work... for comparison with other workflow engines.

I will move the G docs and slides later.

Thanks for the suggestion!

It's built on top of Cadence Temporal(Similar to AWS SWF, Microsoft Azure Durable Function) by expert of Cadence/Temporal to provide clean and simple API/interface for development.

* Easy to understand and get started * Easy to make changes -- no non deterministic errors and no need to write versioning code. * Easy to write and maintain unit tests * Easy to monitor and operate * Provide all power of Cadence/Temporal

Look at the samples for how easy it is, and try it out!