News for hackers. Stuff that matters.
Re:A Bayesian Approach to A/B Testing by noelwelsh on Saturday May 05, 2012 @02:45AM (#3931533)
This way of framing the problem is known as the bandit problem. You can find lots of papers about it (Bayesian and frequentist). As others have mentioned in this thread we have a startup providing bandit algorithms as SaaS: http://mynaweb.com/
Re:A Bayesian Approach to A/B Testing by ced on Saturday May 05, 2012 @07:33PM (#3933622)
I've looked at your website, and from what I gathered, I would make the same criticism as I made for Anscombe's rule: it's not easy at all to decide what rewards should be, and how to put a price on exploration vs. exploitation. The more I think about it, the more I feel that an engineer looking at Beta distributions could weigh the trade-offs and make a better decision than a black-box algorithm with inadequate assumptions.

Granted, this doesn't really scale to testing many combinations of feature, and I think that I can see what you're shooting for. Best of luck with Myna.