Re:Stop A/B Testing and Make Out like a Bandit
by datadon on Monday August 01, 2011 @08:14AM
(#2831722)
Going purely off of your boiled down description, this seems like a more advanced version of Genetify (https://github.com/gregdingle/genetify/wiki).
@noelwelsh - care to comment on now what you're doing at Untyped differs from what Genetify offers?
Re:Stop A/B Testing and Make Out like a Bandit
by e-dard on Monday August 01, 2011 @08:51AM
(#2831853)
The difference in performance between using evolutionary approaches and bandit algorithms is complex and problem dependent. There is no "one algorithm is better than the other".
I've used both approaches for a different application and neither dominate the other.
Re:Stop A/B Testing and Make Out like a Bandit
by StavrosK on Monday August 01, 2011 @08:32AM
(#2831789)
That looks like it uses genetic algorithms, which is much less optimal and more exploratory than bandit algorithms. What noelwelsh is proposing would lead to better results much more quickly and without random permutations of elements.
If I'm right, the idea is that the algorithm dynamically weights the "display frequency" of the two (or n) options. So as one of the a/b options shows itself to be more successful, it's shown more frequently. Because the test is self-correcting, you as an A/B test runner don't have to decide when the results are significant enough, and the program itself will automatically choose the more successful option.