jeffan
We parallelize testing across a massive fleet of browsers/devices so it should actually be much faster than Claude. The open source benchmarks prove it :)
Actually a lot of our mobile users came from Maestro, we hear the maintenance burden is a nightmare.
But to your general point, this benchmark is showing we are finding 3-5X more bugs than the best OSS performance out there right now. We did also try it on tester army e2e and it underperformed Codex + Playwright MCP (this will make it in the benchmark soon).
The "better" comes from custom agents and massively parallelized infrastructure. I'm not sure how many folks want to or even can self-host fleets of GPUs, browsers and Mac minis. We're happy for folks who can but our current offering is for those who want to find the most bugs with the least amount of effort.
But I would say even for medium sized features, it's the same difference between asking your Codex to "review the code make no mistakes" and using a paid AI code review tool.
As your product surface area increases, you need fundamentally different agent architecture to maintain context over time and across bug bash sessions. The benchmark apps are probably among the simplest apps we test on, and even those show a massive perf. gain. For large companies with 1000s of important user flows, have to support different viewports, and need to work across mobile/web/desktop, asking Codex to test it all before a big launch is not scalable.
Even basic constraints like the # of browsers/devices you can run concurrently, the amount of reliable TPM you need, and how many agents can run on a single box become problematic. You also have to think about ktlo: how do you evaluate new model releases in a data driven way, how does agent become more efficient itself over time, etc.
1. Mo generates its own test cases in natural language, but you can ask it to write Playwright and maintain it in your repository if you really want. We actually do have a regression testing product as well where tests are just natural language in your repository. For that product yes there is some vendor lock-in since this is a proprietary format that only our CLI translates into browser actions. But ultimately since it's natural language it's pretty portable :)
2. Mo is focused on exploratory testing and so is inherently non-deterministic. But the regression testing product has a ton of caching built into it that we've written about in the past (not just a single css selector). https://momentic.ai/blog/teaching-browser-agents-user-intent This is a big part of reliability and latency optimization, otherwise your CI would just hang for hours on any reasonable suite.
3. I would say the regression testing product was very much SDET/developer focused since it required you to have a codebase and use Git. Mo is meant to be super easy for anyone to onboard onto and lower the barrier to entry, so we're hoping we'll see more product owners, PMs, QA and even leadership use the product. But devs are always going to be our bread and butter, we ultimately tried to build the product we wanted to use ourselves.
4. We're optimized for recall (# of bugs you find in a given amount of time). The bench shows we find ~3-5X more bugs compared to Codex and Claude Code, with approx. the same precision. The service costs more than the raw harnesses for sure, but that's the all-in cost including our infra, parallel agents, etc.
Everything has to be reproducible and runnable by the general public
1/ General purpose models are still not good at this task. I would personally love it if I could trust my agent when it says its done after a 24h build. But whether it's because of sycophancy, context degradation over time, or something else coding agents still struggle to verify their own work. Maybe with 1B context windows this will change.
2/ Infra is still annoying to setup. We wanted to provide a unified interface across web, mobile and soon desktop as well. So users don't have to worry about provisioning, concurrency, security, etc.
3/ As with any SaaS, even beyond the initial build, maintenance is a headache. I think most companies would rather not consider agentic QA as one of their core competencies and investments. It takes a ton of eng effort to keep the lights on, continually update the models used, train new ones, groom ground truth, onboard infra providers, etc.
That doesn't mean we won't eventually support byok, especially as OSS models catch up! We have our finger on the trigger there.