Re:A/B Testing Significance Calculator
by cheesycheese on Friday April 04, 2014 @03:53PM
(#7533905)
The normal approximation is notoriously [1][2] bad when the binomial proportion you're trying to estimate is small and you don't have a big n. If one still wants a closed-form out-of-the-box formula (to avoid the exact resolution you suggested), one can use the Wilson approximation for instance.
[1]: http://en.wikipedia.org/wiki/Binomial_proportion_confidence_...
[2]: http://projecteuclid.org/download/pdf_1/euclid.ss/1009213286
Re:A/B Testing Significance Calculator
by jessicaraygun on Friday April 04, 2014 @03:36PM
(#7533788)
Hey, I'm the author of Confidence.js which was used to power this A/B testing calculator. That sounds really interesting and worth looking into - thanks for the feedback!
Exact confidence regions can be found, if you do it using Binomial instead.
The question to be asked, if m out of n is the frequency, is for which values of p do you have P(m>=M) = 0.975 and for which value of p do you have P(m<=M) = 0.975, where M ~ Binimial(n,p).
It can be solved easily too.