Re:How Optimizely Almost Got Me Fired (2014)
by cbr on Sunday January 10, 2016 @03:20PM
(#10877002)
With a big enough N, the probability of reaching
significance approaches 1.
Well, not with an A vs A experiment. Then the probability of significance at the p < 0.05 level should stay 5%.
On the other hand, in an A vs B experiment where B is very slightly different from A you're right that with a large enough population you become very likely to reach a p < 0.05 significance level, just with an extremely tiny effect size. The experiment has told you something about the difference between A and B, and you now have high certainty, but it's just such a small difference that you probably don't care.
Have you ever approached a tool in the field that
would fail my proposed test?
Your proposed test being, run an A vs A experiment and a tool is faulty if it concludes there's a difference significant at the 0.05 level more than 5% of the time? All the tools I've used for A/B testing would do fine on this test. This includes a custom tool at a computational advertising startup I worked at, Google's internal experiment platform, and Google Analytics' Content Experiments.
(If I didn't trust a tool to pass this test I'd just use it to get the counts for both sides of the experiment and do that stats on my own.)
On the other hand, in an A vs B experiment where B is very slightly different from A you're right that with a large enough population you become very likely to reach a p < 0.05 significance level, just with an extremely tiny effect size. The experiment has told you something about the difference between A and B, and you now have high certainty, but it's just such a small difference that you probably don't care.
Your proposed test being, run an A vs A experiment and a tool is faulty if it concludes there's a difference significant at the 0.05 level more than 5% of the time? All the tools I've used for A/B testing would do fine on this test. This includes a custom tool at a computational advertising startup I worked at, Google's internal experiment platform, and Google Analytics' Content Experiments.(If I didn't trust a tool to pass this test I'd just use it to get the counts for both sides of the experiment and do that stats on my own.)