Thanks for showing interest!
Here is my best effort to explain what’s happening with the agent, and along the way, I’ll try to answer your questions.
The agent works in two steps: planning and execution.
Planning is essentially the process of converting vague or somewhat shallow and disconnected inputs into a clear instruction set that can be handed over for execution. During development, we realized that converting English directly into an instruction set is a more complex problem for our LLMs at this point, leading to less reliable outputs. To address this, we chose the Gherkin format, which provides structure to the input language. Gherkin focuses on terms like "Given," "When," "Then," and "Should." These words act as high-value clues for the LLMs, resulting in a more coherent instruction set.
Key Points 1) Gherkin Usage: Gherkin is simply a format to assist the planner inside the agent—it’s not enforced as strict syntax. You can write ambiguous statements, and the planner will attempt to resolve them and create the best possible execution plan.
However, if you provide extremely ambiguous statements, the planner might ask for clarification or terminate the execution with a failure condition.
This technology is still in its early stages and will mature with community and customer feedback.
2) Auto-Healing: As the agent reads the page from scratch each time it executes, there are no saved or cached automation steps. It always generates instructions on the fly. Because of this, auto-healing is effectively built-in.
In short, auto-healing is LLM-based feedback. You can refer to the README in our repository, where we explain the process of DOM distillation and credit the reference material.
3) Salesforce Support: We tested multiple scenarios on Salesforce, and the agent was able to handle Salesforce well, with a few exceptions. For instance, in cases of heavy customization where no clear action point exists in the UI (e.g., requiring a user to right-click on a cell and perform some other activity), the DOM distillation process cannot identify the cell effectively.
We are continuously adding more evaluations for Salesforce. At this stage, we’re happy with the results in general. If you have a precise use case, we’d love to understand it better. Feel free to schedule a call, post the use case on our Discord (https://discord.gg/JhSAGhfR55), or create an issue on GitHub.
4) Scaling: Currently, Hercules executes one feature file per run, with all scenarios in the file running sequentially. This is because our research revealed that some users write interdependent scenarios.
For execution across multiple feature files, we can run several containers in parallel. Triggering multiple parallel executions is a DevOps exercise, involving launching containers with different mounts in a cluster or serverless platform.
We also offer this scaling as a paid service—check our pricing page. The agent is free for authoring and running as a single unit, allowing everyone to build awesome solutions around it. However, for large-scale execution, you can contact us.
In terms of stability, we found that execution within a feature file is stable, and running across feature files (as multiple instances) has also worked well in our tests. We’re open to learning more and welcome suggestions from the community and customers.
5) Feature Plans: We’ll update the roadmap soon in the README. Some items are already listed there, but we’d love to see more feature requests on Discord or GitHub so we can prioritize them effectively.
We really appreciate your feedback and are excited about your support!
1) Gherkin Parsing and Execution: How does Hercules handle ambiguous or complex Gherkin steps? For instance, if a test step doesn’t clearly map to a specific UI action, does the agent attempt to infer the intent, or does it request clarification?
2) Auto-Healing Tests: The auto-healing feature sounds incredibly useful. Could you explain how Hercules adapts to changes in the DOM or application logic? Does it rely on LLMs exclusively, or are there fallback mechanisms in place?
3) Salesforce Testing: You mention Salesforce readiness—what specific challenges in Salesforce (like dynamic IDs or Shadow DOMs) does Hercules address?
4) Scaling and CI/CD: How well does Hercules scale for large test suites? Are there any specific limitations to consider when integrating into CI/CD pipelines with high parallelism?
5) Future Plans: Any plans to support non-web-based applications (e.g., mobile or desktop)?
This project looks promising, especially for teams without dedicated QA engineers or those working on fast-evolving platforms. Looking forward to trying it out and seeing the community contributions grow. Kudos to you and the team for making it open-source!