News for hackers. Stuff that matters.
Re:p-Hacking and False Discovery in A/B Testing by cepth on Sunday July 22, 2018 @01:55AM (#17585459)
> clever experimental design and cherry picking can generate many results that are statistically significant at that level.

I’m not sure what’s incorrect about this statement? If you disagree with the “fix” to the problem that is most familiar to me, that’s fine. It’s one of many approaches.

But, at what point did I mislead the parent as to what p-hacking is? What’s your definition?

Re:p-Hacking and False Discovery in A/B Testing by joshuamorton on Sunday July 22, 2018 @02:33AM (#17585543)
p-hacking has nothing to do with any specific significance level. You can p-hack at a significance of .5 or .05 or .000005. A better definition would be

>p-hacking is a set of related techniques, whereby clever experimental design and cherry picking of data can generate results that falsely appear statistically significant.

There are a few important differences here:

1. The effect is not statistically significant. In fact often, there is no effect at all.

2. There is no mention of a specific significance level.

Those are both important.