Re:A Bayesian Approach to A/B Testing
by noelwelsh on Saturday May 05, 2012 @02:45AM
(#3931533)
This way of framing the problem is known as the bandit problem. You can find lots of papers about it (Bayesian and frequentist). As others have mentioned in this thread we have a startup providing bandit algorithms as SaaS: http://mynaweb.com/
[ Parent
| on HN ]
I've looked at your website, and from what I gathered, I would make the same criticism as I made for Anscombe's rule: it's not easy at all to decide what rewards should be, and how to put a price on exploration vs. exploitation. The more I think about it, the more I feel that an engineer looking at Beta distributions could weigh the trade-offs and make a better decision than a black-box algorithm with inadequate assumptions.
Granted, this doesn't really scale to testing many combinations of feature, and I think that I can see what you're shooting for. Best of luck with Myna.