MGabrielDenton

Can confirm... not fun

Hey, I'm Gabe (one of the engineers @Momentic) and yes that is possible. The Momentic cli is made for running e2e tests with natural language locators, assertions, and some other features so that you don't have to worry about small element changes failing your tests. Mo will also be able to update these and add new coverage upon your request.

You're right stock CC was pretty terrible. Thats why we compared against CC && Codex with agent browser and playwright mcp as well as devin. Do you have any other agents / configurations you'd like to see?

We had someone recommend browser-use and are running it against the benchmark currently. We're hosting the results here if you'd like to see more: https://momentic.ai/qa-bench

Agreed! We've also found that the tooling you use makes a big difference in how efficient and precise the agent is, which is why we included the comparison of Playwright MCP vs agent-browser. When we were first starting Mo we tried using several different browser interfaces but we weren't able to get the recall we wanted. We settled on rewriting some custom tooling based on our mcp for our investigatory/exploratory agents. Tooling about how to interact with the A11y and DOM, breaking out easy CUA subproblems, and browser infra are still all non-trivial problems.

Thats awesome, there's tons of cool work being done in exploratory testing right now. We'd love to see how it stacks up against Codex/CC/Mo. If you want to be added to the bench feel free to send over the results/how to repro w/ a pr on qa-bench and we can merge it!

It would be really nice to be able to plug in the subs from the labs. The industry seems to be moving that way even if its still super limited. Ex https://loginwith.ai

Good question, in short our harness & e2e tests act as compliments with the frontier rather than competitors. Architectural choices and data you provide really matter for domain specific tasks. Mo will be working with harness specific tooling like: previous momentic test results, our KB, and eventually creating new Momentic regression. Having smarter/more capable models will improve the efficiency of AI generated tests (coverage/runtime) and ideally do less repetition in exploratory testing.

Hey I'm Gabe (one of the engineers @Momentic). In general Mo's advantage is in the total amount of bugs found. In comparison to claude code we find roughly 5x as many bugs at 35% less per bug (for the 60 min session).