Re:p-Hacking and False Discovery in A/B Testing
by joshuamorton on Sunday July 22, 2018 @02:33AM
(#17585543)
p-hacking has nothing to do with any specific significance level. You can p-hack at a significance of .5 or .05 or .000005. A better definition would be
>p-hacking is a set of related techniques, whereby clever experimental design and cherry picking of data can generate results that falsely appear statistically significant.
There are a few important differences here:
1. The effect is not statistically significant. In fact often, there is no effect at all.
2. There is no mention of a specific significance level.
Those are both important.
I’m not sure what’s incorrect about this statement? If you disagree with the “fix” to the problem that is most familiar to me, that’s fine. It’s one of many approaches.
But, at what point did I mislead the parent as to what p-hacking is? What’s your definition?