Re:It’s not just p=0.048 vs. p=0.052
by krona on Wednesday September 11, 2019 @07:49AM
(#20938943)
Well, if there is no effect (the effect-size is zero), two different experimenters will likely see two different p-values regardless of how large an experiment either of them runs.
[ Parent
| on HN ]
Re:It’s not just p=0.048 vs. p=0.052
by mehrdadn on Wednesday September 11, 2019 @07:51AM
(#20938957)
So by this logic "it is completely consistent with the null hypothesis to see p-values of 0.00000001 and 0.99999999 from two replications of the same damn experiment"?
At which point, what is even the point of this statement?
Re:It’s not just p=0.048 vs. p=0.052
by penagwin on Wednesday September 11, 2019 @09:27AM
(#20939880)
My understanding (Someone please correct me if I'm wrong) is that while the values can range from limx->0 and limx->1 , the actual distribution will be skewed in cases where it is statistically significant. E.g. If you repeat the experiment 100x you could find a p=.999999 but more likely they'll be close to .05 (or whatever alpha you choose)
If the null hypothesis is true the p-value is distributed uniformly in [0 1] (at least ignoring discrete data, composite hypothesis and other cases which are not easy).
The distribution will be different when then true hypothesis is not true (but you may also get non-significant results even if the null hypothesis is not true).
I’m not sure if that’s what you mean by “the actual distribution will be skewed in cases where it is statistically significant.”
Re:It’s not just p=0.048 vs. p=0.052
by penagwin on Wednesday September 11, 2019 @01:02PM
(#20942301)
Sorry I was definitely kinda vague - I meant when the alternative hypothesis is true the distribution should be skewed towards 0.
Re:It’s not just p=0.048 vs. p=0.052
by lonelappde on Wednesday September 11, 2019 @08:57AM
(#20939567)
The point is that you need to replicate experiments and not cherry pick the one with the highest p-value.