News for hackers. Stuff that matters.
Re:p-Hacking and False Discovery in A/B Testing by generallyfalse on Tuesday July 24, 2018 @09:22AM (#17600733)
Caveat emperator, I am reading the first pages of the article you link to. In page 1519 they say 1-alpha is the desired significance level. This is wrong, perhaps they mean that alpha is the significance level and 1-alpha is the desired confidence level. In step 3 they say: Preferred test statistics are the ones that can control Type I errors. But that is wrong, Type I error is a parameter you fix so it is not related to the test statistic. Later giving examples of uniform most powerful statistics they require data following a normal distribution, but in web data the distribution can be a mixture of normals whose means depend of the hour. So perhaps the examples are not realistic in the web setting. To be continued.

There are no comments on this comment yet.