News for hackers. Stuff that matters.
Re:It’s not just p=0.048 vs. p=0.052 by Symmetry on Wednesday September 11, 2019 @09:23AM (#20939834)
Put a Number on It! did a piece a while ago going through some psychology pieces that were part of a replication effort. They found that half failed to replicate but that people in a betting market could often tell which ones were going to replicate or not. The author also did a blind test himself and was also able to guess which ones would replicate. He laid out several rules of thumb, most significantly to the article Jacob’s Rule of Anti-Significance: A result with a p-value just above 0.05 could well be true. A result with a p-value just below 0.05 is almost certainly false.

More importantly, p=0.06 means that the researchers are honest. They could have easily p-hacked the results below 0.05 but chose not to. The opposite is true when p=0.049.

https://putanumonit.com/2018/09/07/the-scent-of-bad-psycholo...

Re:It’s not just p=0.048 vs. p=0.052 by UncleMeat on Wednesday September 11, 2019 @11:54AM (#20941512)
This is true in more than just psych. A strategy for systems papers is to select the system that is second on the graphs. The author's results are not trustworthy, but the second place system was usable and ran reasonably well in the hands of somebody other than the original author.
Re:It’s not just p=0.048 vs. p=0.052 by hadsed on Wednesday September 11, 2019 @02:14PM (#20943062)
For ML practitioners out there, this is a great method for a field also in a replication crisis.
Re:It’s not just p=0.048 vs. p=0.052 by WengerPen on Wednesday September 11, 2019 @09:29AM (#20939902)
That is a pretty interesting conclusion, do you know whether his work has been replicated to confirm the outcomes?
Re:It’s not just p=0.048 vs. p=0.052 by targonca on Wednesday September 11, 2019 @10:58AM (#20940855)
Yep, it's confirmed with p = 0.049