daniiel

Not sure precision/recall is the number I'd look at first, because the thing I'd actually want to know is how often the tool says pass on a flow that is already broken, and a silent pass is worse than a flaky test, since the flaky one at least makes some noise.

I'm building something in the same space and this bit me quite early, so what I ended up doing was basically mutation testing: take a handful of open source apps, plant bugs on purpose, and count how many of them the tool waves through anyway. Seems that the pass/fail counts were not the interesting part at all, the interesting part was letting the engine answer with a third option, "I can't tell", because once that was allowed the false passes mostly disappeared, it simply stopped guessing. I would say a guessed pass is worse than having no test, and I keep every run replayable for the same reason, otherwise I wouldn't trust my own numbers either.

Curious how Mo (nice name) handles the middle ground - the docs say a bug it can't reproduce is not reported as confirmed, but does it show up anywhere or is it just dropped? And for QA Bench, did you plant bugs yourselves or only label existing ones?