Re:20 lines of code that beat A/B testing every time
by timr on Wednesday May 30, 2012 @12:23AM
(#4041367)
"Stopping a test when you reach a "statistically significant" result is the wrong way to do A/B testing."
Nobody said that it was. But when you do regular split testing, you can use power analysis to estimate the length of time you need to run an experiment to get a significant result at a certain precision:
Nobody said that it was. But when you do regular split testing, you can use power analysis to estimate the length of time you need to run an experiment to get a significant result at a certain precision:
http://en.wikipedia.org/wiki/Statistical_power
You can't do this (at least, not easily) when you're using bandit models, because none of the assumptions are valid.