- Precision/recall vs pass/fail: I really like this point, and I would say QA bench is actually not a great fit for measuring pass/fail currently, because all of the fixtures have numerous bugs in them, and almost all agent runs find at least one (so the fail rate is basically zero). My assumption would be that recall is correlated with pass/fail accuracy, but we are working on a PR-review variant of QA-bench that I think would be more suited to measuring pass/fail directly.
- I think the insight about allowing "unsure" is interesting, and this is actually similar to what we have for some of the completions in regular Momentic tests, although I'm curious how you generate something useful for the user from the agent basically saying "there might be a bug in this code but I don't know what it is."
- Internally, if Mo can't reproduce a suspected bug it gets discarded. Although we do review some of these to make sure that there aren't issues with the reproduction agent.
- QA bench has zero planted bugs; they're all from the original source. Initially, we did try planting bugs and even generating apps with given bugs using an LLM, but we found that this didn't actually help us establish the ground truth because the agents were still finding bugs that we hadn't planted and needed manual review.
I'm building something in the same space and this bit me quite early, so what I ended up doing was basically mutation testing: take a handful of open source apps, plant bugs on purpose, and count how many of them the tool waves through anyway. Seems that the pass/fail counts were not the interesting part at all, the interesting part was letting the engine answer with a third option, "I can't tell", because once that was allowed the false passes mostly disappeared, it simply stopped guessing. I would say a guessed pass is worse than having no test, and I keep every run replayable for the same reason, otherwise I wouldn't trust my own numbers either.
Curious how Mo (nice name) handles the middle ground - the docs say a bug it can't reproduce is not reported as confirmed, but does it show up anywhere or is it just dropped? And for QA Bench, did you plant bugs yourselves or only label existing ones?