(I suspect you could get pretty far with a Monte Carlo simulation, and that would let you bypass most of the math anyway.)
https://github.com/Ealdwulf/BBChop/blob/master/BBChop/doc/Ba...
(Edit- had wrong link originally)
When gathering more evidence, you'd use your new belief about which cookie bowl Fred has as P(H1)=0.6 and P(H2)=0.4
I was specifically interested in the application to binary search / bisection in the presence of flaky tests.
Here's a nice presentation on an analysis of the method, references start on slide 47
https://people.orie.cornell.edu/pfrazier/Presentations/2014....
Paraphrasing the theorem from Jedynak, Frazier, Sznitman (2011):
Suppose [flake probability] is constant, known, and bounded away from 1/2, and we use the entropy loss function. ... The policy that chooses [next test point] at the median of [current posterior distribution] is optimal.
And some python code that implements a version of the analysis
https://github.com/choderalab/thresholds/blob/master/thresho...
You do have to provide flake rate probability to do the probability estimates, but even roughly correct rates work fine. Running bisects assuming a 5% chance of getting a false positive or negative barely adds more steps and greatly improves robustness.
The math is pretty simple too-- my old prototype might still be in Google's monorepo; I should reimplement it for the open source world.