News for hackers. Stuff that matters.
Re:Statistical significance & other A/B test pitfalls by sesqu on Friday July 30, 2010 @10:19PM (#1563099)
You're right, I did have the wrong scale, and should have realized. I believe this plot to be correct. Black is binomial, red is normal, both for the difference in conversion rates.

http://img267.imageshack.us/img267/6691/binnormdifflikelihoo...

The normal approximation seems reasonable, and so a t-test shouldn't cause problems. I made a plot of the relative error of the normal approximation; the tails are indeed too fat. In particular, there's a bump to the left of 0, so a t-test would slightly overestimate p.

http://img205.imageshack.us/img205/6691/binnormdifflikelihoo...

Re:Statistical significance & other A/B test pitfalls by btilly on Saturday July 31, 2010 @11:07AM (#1563796)
The problem is that when you're near confidence you're in the tail, and the student t-test is extremely sensitive to the shape of said tail. If n is large enough this difference will be washed away as everything converges to normals. But with smaller sample sizes the difference can be quite significant.