Re:Redefine statistical significance
by jjoonathan on Thursday September 07, 2017 @01:02PM
(#15193719)
> It doesn't stop p-hacking, it just makes it harder.
I could see the reverse happening, where higher p-value standards lead to normalization of deviance in the form of worse p-hacking.
Re:Redefine statistical significance
by felippee on Thursday September 07, 2017 @01:27PM
(#15193942)
I agree with that. In my prediction legitimate, honest research with p~0.05 will become unpublishable, while p-hacked bullshit will prevail in glory.
Re:Redefine statistical significance
by PeterisP on Thursday September 07, 2017 @04:39PM
(#15195496)
If you do a honest preliminary study and get a p~0.05 result, then repeating the study with 70% larger sample size should get you p~0.005; but if you've p-hacked, it won't.
Re:Redefine statistical significance
by felippee on Thursday September 07, 2017 @04:58PM
(#15195648)
But sometimes getting 70% larger sample will cost you so much money, that you'd need another grant, one you'd get after you'd publish your p=0.05 result, which given the new rules you'd not be able to publish...
Statistics is just another method and almost every method can be hacked or abused. Science is not about putting checkmarks in tables but about reading and understanding ideas and reproducing the results. Tweaking some numeric values is not going to help the review process which is fundamentally broken these days.
On the other hand, you might expect that new discoveries by nature have less data since data is likely more expensive for brand new research, and by extension a lower likelihood of meeting these sorts of stringent statistical requirements. Decreasing the p-value threshold may be counter-productive if we dismiss legitimate new discoveries due to essentially economic constraints with data gathering, which would have the impact of making it less likely to get funding to pursue the problem in more depth, thereby slowing the advance of discoveries.