News for hackers. Stuff that matters.
Re:A Bayesian Approach to A/B Testing by zeroonetwothree on Friday May 04, 2012 @01:42PM (#3929468)
There's a related notion of an "adaptive" statistical design, where the allocation of users to each test group varies based on prior performance of the group. For example, if after the first 100 users you notice that group A seems to be doing slightly better than group B, you will favor it by allocation more users to that group. You can compute this allocation in such a way as to maximize the number of successes. In particular it will converge to always picking the better approach eventually, assuming there is a real difference. This also means you don't really need to "stop" the experiment to make everyone use the better version (although you may want to for other reasons).

Here is one paper: http://web.eecs.umich.edu/~qstout/pap/SciProg00.pdf

Re:A Bayesian Approach to A/B Testing by aaronjg on Friday May 04, 2012 @01:58PM (#3929539)
Noel Welsh at untyped has been working on a cool implementation of this adaptive design.

Check it out http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-m...

And the HN discussion: http://news.ycombinator.com/item?id=3867380