Re:ABBA: A/B Test (Split Test) Calculator
by connoredel on Thursday June 23, 2016 @01:52AM
(#11959110)
This is in my opinion the best simple A/B test significance calculator. I like it for two reasons:
1) The multiple testing correction, which many people ignore when testing > 1 variants vs. the control
2) They display not only the p-value, but the confidence interval around the _difference_ between the variants (i.e. the effect size). It's tempting to observe an increase of say 50% that is statistically significant and think that you can be confident about the 50%. This is not the case! You chose your sample size to be confident that the actual effect size is > 0 -- not that you would have a tight confidence interval around whatever difference you observed. Depending on your sample, the lower bound could still be close to 0.
1) The multiple testing correction, which many people ignore when testing > 1 variants vs. the control
2) They display not only the p-value, but the confidence interval around the _difference_ between the variants (i.e. the effect size). It's tempting to observe an increase of say 50% that is statistically significant and think that you can be confident about the 50%. This is not the case! You chose your sample size to be confident that the actual effect size is > 0 -- not that you would have a tight confidence interval around whatever difference you observed. Depending on your sample, the lower bound could still be close to 0.