News for hackers. Stuff that matters.
Re:No Paper Is That Good by sarbaz on Friday July 13, 2018 @08:17AM (#17522238)
A lot of this is simply that p < 0.05 is a very low bar for research. If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about. Without this thing, it's not science.

A good mechanism that accurately explains some part of the world can be validated with a much better result than p < 0.05, because it can be properly isolated from other factors and its effect size can be strengthened.

Without a mechanism to explain how things work, the experimental results that get published are probably just noise. And because p < 0.05 is such a low bar to pass, lots of noise gets published. Trying to reproduce an experiment whose results were drawn from /dev/urandom is pointless.

In fact I'm surprised the replication was so high. I was expecting it to be around 5% (i.e. results are totally random). So maybe there is some hidden merit to all of this

Re:No Paper Is That Good by nicoburns on Friday July 13, 2018 @08:27AM (#17522311)
> If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about.

This, 1000 times over. Unfortunately we seem to teach that statistical analysis is sufficient to infer causation in a lot our undergraduate science programs.

Re:No Paper Is That Good by traverseda on Friday July 13, 2018 @08:31AM (#17522338)
I wouldn't expect totally random because people have common sense. I'd expect it to be similar to the folk-wisdom of the time, since a lot of it is going to be repackaging of that same folk-wisdom, and negative results tend not to get published, or get massaged until they're positive.

I'd expect social-psychology to mirror the dominant opinions of people in other segments of the humanities, although it is of course hard to determine the direction of that relationship.