Re:p-Hacking and False Discovery in A/B Testing
by generallyfalse on Tuesday July 24, 2018 @09:22AM
(#17600733)
Caveat emperator, I am reading the first pages of the article you link to. In page 1519 they say 1-alpha is the desired significance level. This is wrong, perhaps they mean that alpha is the significance level and 1-alpha is the desired confidence level. In step 3 they say: Preferred test statistics are the ones that can control Type I errors. But that is wrong, Type I error is a parameter you fix so it is not related to the test statistic. Later giving examples of uniform most powerful statistics they require data following a normal distribution, but in web data the distribution can be a mixture of normals whose means depend of the hour. So perhaps the examples are not realistic in the web setting. To be continued.
[ Parent
| on HN ]
There are no comments on this comment yet.