News for hackers. Stuff that matters.
Re:A/B Testing Duration Data by shalmanese on Sunday August 25, 2013 @02:51PM (#6273181)
The t-test is invalid because duration data is non-normal. You can actually see duration data as a histogram in GA if you want to test hypotheses on it's distribution.
Re:A/B Testing Duration Data by btilly on Sunday August 25, 2013 @03:00PM (#6273215)
I came here to say this, and am glad someone else said it first.

That on top of the unwarranted distribution assumption is very dangerous. How dangerous? If the true distribution has heavier tails than assumed, then the true variance can be many times what is assumed. If the true variance is, say, 4x what is assumed then an average standard deviation variation will show up as passing a 95% confidence interval in a random direction. Getting to 99% confidence by accident becomes quite easy, with bad results.

Re:A/B Testing Duration Data by tel on Sunday August 25, 2013 @03:14PM (#6273259)
Ugh. I didn't even read the article and was looking for this response. Whenever I see "A/B test" I am usually right in assuming that the author is about to assume normality.
Re:A/B Testing Duration Data by btilly on Sunday August 25, 2013 @05:20PM (#6273780)
I understand, but the author of this particular article actually does know better and usually gets his statistics right.
Re:A/B Testing Duration Data by jimmybot on Sunday August 25, 2013 @06:37PM (#6274048)
Can you be more specific on why you think the t-test is invalid when the data is non-normal? Duration data may be non-normal, but given finite visits and a reasonable number of visits, sampling for the mean surely is.
Re:A/B Testing Duration Data by btilly on Monday August 26, 2013 @09:09AM (#6276770)
You are right that in the presence of sufficient data, everything converges to the normal and we're fine.

However the difference between the t-test and the normal to which it is an approximation is that the t-test takes into account corrections the rate of convergence to the estimated normal. The details of that convergence is dependent upon the distribution of individual samples, and the t-test is predicated on the assumption that each and every sample is from a normal distribution.

Re:A/B Testing Duration Data by saosebastiao on Sunday August 25, 2013 @03:12PM (#6273256)
Not only that, but it appears the sample sizes are not equivalent.
Re:A/B Testing Duration Data by bobbinsIII on Sunday August 25, 2013 @05:45PM (#6273847)
you can still use a t-test for non-normal data...