P < 0.05 Considered Harmful

Re:P < 0.05 Considered Harmful by ramblenode on Monday April 10, 2023 @09:07PM (#35519387)

I think part of the problem with p-values and NHST is that it encourages (or doesn't discourage) underpowered studies. That's because p-hacking benefits from the noise of underpowered studies. If you can test a large number of models and only report the significant one then an underpowered study with high type I error rate gives you a greater chance of a significant result.

So I think you are correct that properly powering studies is the crucial thing, but the incentives are against fixing this as long as lone p-values are publishable.

[ Parent | on HN ]

Re:P < 0.05 Considered Harmful by spekcular on Monday April 10, 2023 @09:57PM (#35519763)

But here the issue is the uncorrected multiple testing and under-reporting of results, not the p-values themselves. Any criterion for judging the presence of an effect is going to suffer from the same issue, if researchers don't pre-register and report all of their analyses (since otherwise you have censored data, "researcher degrees of freedom," and so on). This is really a a problem with the design and reporting of studies, not the analysis method.

[ Parent | on HN ]