Re:Out of the cesspool and into the sewer: A/B testing trap
by tel on Monday June 21, 2010 @11:36AM
(#1449086)
Here's a thought: apply successive A/B testing in steps similarly to the Metropolis-Hastings algorithm. The core sys that for each new test (each new "jump") you evaluate the probability ratio a = p_new/p_old where p_x is the probability that x is successful. If a is greater than 1, go with the new, if it's less than 1, still go with the new a% of the time.
By implementing parts of this algorithm you might be able to generate a random walk scenario that is local minima resistant.
By implementing parts of this algorithm you might be able to generate a random walk scenario that is local minima resistant.