Show HN: Hercules – Open-Source software testing agent
After spending way too many late nights wrestling with brittle test scripts and UIs that seem to change just to spite me, we decided to do something about it. So we built Hercules, an open-source testing agent aiming to make end-to-end testing of modern web applications less of a Herculean task (pun absolutely intended).
P.S.- Paid solutions can cost upwords of $15000 and still struggle with maintenance.
What is Hercules?
Hercules is an AI-powered testing agent that turns simple Gherkin steps into fully automated end-to-end tests, eliminating the need for coding skills. It leverages Large Language Models (LLMs) to reason and perform actions based on test requirements.
Key Features-
- Gherkin In, Results Out: Just provide your tests in Gherkin format, and Hercules executes them automatically, outputting results in JUnit XML and HTML reports. - Free and Open Source: Hercules is released under the AGPL v3 license, allowing anyone to use, modify, and contribute. - Salesforce Ready: Capable of handling intricate UIs, Hercules excels in testing platforms like Salesforce. - No Code Required: Automates Gherkin features without the need for scripting or dealing with selectors. - Multilingual Support: Supports multiple languages out of the box, facilitating global collaboration. - Precisely Accurate: Records execution videos and captures network logs for reliable test results. - No Maintenance: Features auto-healing capabilities to adapt to changes without manual intervention. - CI/CD Ready: Easily integrates into your CI/CD pipeline and can run in Docker containers.
Why is Hercules Different?
Unlike traditional testing tools, Hercules acts as an intelligent agent that can think, reason, and adapt. It's built on a multi-agent architecture, allowing it to plan and execute tasks based on the provided Gherkin steps. Additionally, it comes packed with tool usage like Browsers, or APIs; and users can build/attach external tools too.
Try It Out
The source code and detailed documentation are available on GitHub: [https://github.com/test-zeus-ai/testzeus-hercules](https://g...
To get started:
- Install via PyPI:
```bash
pip install testzeus-hercules
playwright install --with-deps
```
```bash
docker pull testzeus/hercules:latest
```
Feedback Welcome-
We'd love to hear your thoughts, experiences, or even horror stories about testing gone wrong. Feel free to open issues on GitHub or join forces on our Discord : https://discord.gg/4fyEMWVD
1) Gherkin Parsing and Execution: How does Hercules handle ambiguous or complex Gherkin steps? For instance, if a test step doesn’t clearly map to a specific UI action, does the agent attempt to infer the intent, or does it request clarification?
2) Auto-Healing Tests: The auto-healing feature sounds incredibly useful. Could you explain how Hercules adapts to changes in the DOM or application logic? Does it rely on LLMs exclusively, or are there fallback mechanisms in place?
3) Salesforce Testing: You mention Salesforce readiness—what specific challenges in Salesforce (like dynamic IDs or Shadow DOMs) does Hercules address?
4) Scaling and CI/CD: How well does Hercules scale for large test suites? Are there any specific limitations to consider when integrating into CI/CD pipelines with high parallelism?
5) Future Plans: Any plans to support non-web-based applications (e.g., mobile or desktop)?
This project looks promising, especially for teams without dedicated QA engineers or those working on fast-evolving platforms. Looking forward to trying it out and seeing the community contributions grow. Kudos to you and the team for making it open-source!
Thanks for showing interest!
Here is my best effort to explain what’s happening with the agent, and along the way, I’ll try to answer your questions.
The agent works in two steps: planning and execution.
Planning is essentially the process of converting vague or somewhat shallow and disconnected inputs into a clear instruction set that can be handed over for execution. During development, we realized that converting English directly into an instruction set is a more complex problem for our LLMs at this point, leading to less reliable outputs. To address this, we chose the Gherkin format, which provides structure to the input language. Gherkin focuses on terms like "Given," "When," "Then," and "Should." These words act as high-value clues for the LLMs, resulting in a more coherent instruction set.
Key Points 1) Gherkin Usage: Gherkin is simply a format to assist the planner inside the agent—it’s not enforced as strict syntax. You can write ambiguous statements, and the planner will attempt to resolve them and create the best possible execution plan.
However, if you provide extremely ambiguous statements, the planner might ask for clarification or terminate the execution with a failure condition.
This technology is still in its early stages and will mature with community and customer feedback.
2) Auto-Healing: As the agent reads the page from scratch each time it executes, there are no saved or cached automation steps. It always generates instructions on the fly. Because of this, auto-healing is effectively built-in.
In short, auto-healing is LLM-based feedback. You can refer to the README in our repository, where we explain the process of DOM distillation and credit the reference material.
3) Salesforce Support: We tested multiple scenarios on Salesforce, and the agent was able to handle Salesforce well, with a few exceptions. For instance, in cases of heavy customization where no clear action point exists in the UI (e.g., requiring a user to right-click on a cell and perform some other activity), the DOM distillation process cannot identify the cell effectively.
We are continuously adding more evaluations for Salesforce. At this stage, we’re happy with the results in general. If you have a precise use case, we’d love to understand it better. Feel free to schedule a call, post the use case on our Discord (https://discord.gg/JhSAGhfR55), or create an issue on GitHub.
4) Scaling: Currently, Hercules executes one feature file per run, with all scenarios in the file running sequentially. This is because our research revealed that some users write interdependent scenarios.
For execution across multiple feature files, we can run several containers in parallel. Triggering multiple parallel executions is a DevOps exercise, involving launching containers with different mounts in a cluster or serverless platform.
We also offer this scaling as a paid service—check our pricing page. The agent is free for authoring and running as a single unit, allowing everyone to build awesome solutions around it. However, for large-scale execution, you can contact us.
In terms of stability, we found that execution within a feature file is stable, and running across feature files (as multiple instances) has also worked well in our tests. We’re open to learning more and welcome suggestions from the community and customers.
5) Feature Plans: We’ll update the roadmap soon in the README. Some items are already listed there, but we’d love to see more feature requests on Discord or GitHub so we can prioritize them effectively.
We really appreciate your feedback and are excited about your support!
With Ollama in our machines, we didn't get good results maybe because of model limitation on function calling.
We are experimenting with more models and smaller models so that running local with local models becomes possible.
Although if you check the readme and example config file there is a way to give it a try with ollama.
I also suggest that you join our slack group so that we can discuss this in one on one and try to solve this for your case. https://join.slack.com/t/testzeuscommunityhq/shared_invite/z...
lemme know if/when you build the CLI agent that write my gherkin too
Checkout our slack channel, we can discuss these ideas in the forum, I would love you hear about your feedback and experience with the runs.
https://join.slack.com/t/testzeuscommunityhq/shared_invite/z...