shriyansha

Wow, you read our minds.

Checkout our slack channel, we can discuss these ideas in the forum, I would love you hear about your feedback and experience with the runs.

https://join.slack.com/t/testzeuscommunityhq/shared_invite/z...

Deepak I would suggest give it a try check this https://github.com/test-zeus-ai/testzeus-hercules/blob/main/...

I also suggest that you join our slack group so that we can discuss this in one on one and try to solve this for your case. https://join.slack.com/t/testzeuscommunityhq/shared_invite/z...

We tried Gpt-4o, Anthropic haiku, Groq's 8b tool preview, Llama 3.2 8b tool support via Groq and all worked well.

With Ollama in our machines, we didn't get good results maybe because of model limitation on function calling.

We are experimenting with more models and smaller models so that running local with local models becomes possible.

Although if you check the readme and example config file there is a way to give it a try with ollama.

Hi Ajith,

Thanks for showing interest!

Here is my best effort to explain what’s happening with the agent, and along the way, I’ll try to answer your questions.

The agent works in two steps: planning and execution.

Planning is essentially the process of converting vague or somewhat shallow and disconnected inputs into a clear instruction set that can be handed over for execution. During development, we realized that converting English directly into an instruction set is a more complex problem for our LLMs at this point, leading to less reliable outputs. To address this, we chose the Gherkin format, which provides structure to the input language. Gherkin focuses on terms like "Given," "When," "Then," and "Should." These words act as high-value clues for the LLMs, resulting in a more coherent instruction set.

Key Points 1) Gherkin Usage: Gherkin is simply a format to assist the planner inside the agent—it’s not enforced as strict syntax. You can write ambiguous statements, and the planner will attempt to resolve them and create the best possible execution plan.

However, if you provide extremely ambiguous statements, the planner might ask for clarification or terminate the execution with a failure condition.

This technology is still in its early stages and will mature with community and customer feedback.

2) Auto-Healing: As the agent reads the page from scratch each time it executes, there are no saved or cached automation steps. It always generates instructions on the fly. Because of this, auto-healing is effectively built-in.

In short, auto-healing is LLM-based feedback. You can refer to the README in our repository, where we explain the process of DOM distillation and credit the reference material.

3) Salesforce Support: We tested multiple scenarios on Salesforce, and the agent was able to handle Salesforce well, with a few exceptions. For instance, in cases of heavy customization where no clear action point exists in the UI (e.g., requiring a user to right-click on a cell and perform some other activity), the DOM distillation process cannot identify the cell effectively.

We are continuously adding more evaluations for Salesforce. At this stage, we’re happy with the results in general. If you have a precise use case, we’d love to understand it better. Feel free to schedule a call, post the use case on our Discord (https://discord.gg/JhSAGhfR55), or create an issue on GitHub.

4) Scaling: Currently, Hercules executes one feature file per run, with all scenarios in the file running sequentially. This is because our research revealed that some users write interdependent scenarios.

For execution across multiple feature files, we can run several containers in parallel. Triggering multiple parallel executions is a DevOps exercise, involving launching containers with different mounts in a cluster or serverless platform.

We also offer this scaling as a paid service—check our pricing page. The agent is free for authoring and running as a single unit, allowing everyone to build awesome solutions around it. However, for large-scale execution, you can contact us.

In terms of stability, we found that execution within a feature file is stable, and running across feature files (as multiple instances) has also worked well in our tests. We’re open to learning more and welcome suggestions from the community and customers.

5) Feature Plans: We’ll update the roadmap soon in the README. Some items are already listed there, but we’d love to see more feature requests on Discord or GitHub so we can prioritize them effectively.

We really appreciate your feedback and are excited about your support!