Re:Statistical significance & other A/B test pitfalls
by btilly on Saturday July 31, 2010 @11:07AM
(#1563796)
The problem is that when you're near confidence you're in the tail, and the student t-test is extremely sensitive to the shape of said tail. If n is large enough this difference will be washed away as everything converges to normals. But with smaller sample sizes the difference can be quite significant.
http://img267.imageshack.us/img267/6691/binnormdifflikelihoo...
The normal approximation seems reasonable, and so a t-test shouldn't cause problems. I made a plot of the relative error of the normal approximation; the tails are indeed too fat. In particular, there's a bump to the left of 0, so a t-test would slightly overestimate p.
http://img205.imageshack.us/img205/6691/binnormdifflikelihoo...