Re:Launch HN: Momentic (YC W24) – Agents that catch bugs by ...
by momentichenry on Monday November 02, 2026 @07:15PM
(#50346213)
Hey there! This is Henry, one of the engineers working on Mo. Thanks for reading our post! A lot of interesting stuff here:
- Precision/recall vs pass/fail: I really like this point, and I would say QA bench is actually not a great fit for measuring pass/fail currently, because all of the fixtures have numerous bugs in them, and almost all agent runs find at least one (so the fail rate is basically zero). My assumption would be that recall is correlated with pass/fail accuracy, but we are working on a PR-review variant of QA-bench that I think would be more suited to measuring pass/fail directly.
- I think the insight about allowing "unsure" is interesting, and this is actually similar to what we have for some of the completions in regular Momentic tests, although I'm curious how you generate something useful for the user from the agent basically saying "there might be a bug in this code but I don't know what it is."
- Internally, if Mo can't reproduce a suspected bug it gets discarded. Although we do review some of these to make sure that there aren't issues with the reproduction agent.
- QA bench has zero planted bugs; they're all from the original source. Initially, we did try planting bugs and even generating apps with given bugs using an LLM, but we found that this didn't actually help us establish the ground truth because the agents were still finding bugs that we hadn't planted and needed manual review.
- Precision/recall vs pass/fail: I really like this point, and I would say QA bench is actually not a great fit for measuring pass/fail currently, because all of the fixtures have numerous bugs in them, and almost all agent runs find at least one (so the fail rate is basically zero). My assumption would be that recall is correlated with pass/fail accuracy, but we are working on a PR-review variant of QA-bench that I think would be more suited to measuring pass/fail directly.
- I think the insight about allowing "unsure" is interesting, and this is actually similar to what we have for some of the completions in regular Momentic tests, although I'm curious how you generate something useful for the user from the agent basically saying "there might be a bug in this code but I don't know what it is."
- Internally, if Mo can't reproduce a suspected bug it gets discarded. Although we do review some of these to make sure that there aren't issues with the reproduction agent.
- QA bench has zero planted bugs; they're all from the original source. Initially, we did try planting bugs and even generating apps with given bugs using an LLM, but we found that this didn't actually help us establish the ground truth because the agents were still finding bugs that we hadn't planted and needed manual review.