News for hackers. Stuff that matters.
Re:No Paper Is That Good by traverseda on Friday July 13, 2018 @07:37AM (#17522018)
On the psychology replication crisis

>Overall, 36% of the replications yielded significant findings (p value below 0.05) compared to 97% of the original studies that had significant effects. The mean effect size in the replications was approximately half the magnitude of the effects reported in the original studies.

https://en.wikipedia.org/wiki/Replication_crisis#Psychology_...

Of course if you point out that psychology research is strongly biased, the immediate response is that you must be an anti-intellectual. As if the system that produced such misleading results is a representation of all intellectual pursuits, and not just a mistake. The same way you get called an anti-intellectual for questioning the post-modern-analyses of whatever.

I'm really sick of people claiming that criticizing obviously bad science is "anti-intellectual". When you produce that many useless papers you're obviously, as a field, lacking an understanding of why science is good and useful in the first place.

I'm presuming that economics has similar problems with replication, and that you can only trust the most basic and obvious of their findings.

Re:No Paper Is That Good by sarbaz on Friday July 13, 2018 @08:17AM (#17522238)
A lot of this is simply that p < 0.05 is a very low bar for research. If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about. Without this thing, it's not science.

A good mechanism that accurately explains some part of the world can be validated with a much better result than p < 0.05, because it can be properly isolated from other factors and its effect size can be strengthened.

Without a mechanism to explain how things work, the experimental results that get published are probably just noise. And because p < 0.05 is such a low bar to pass, lots of noise gets published. Trying to reproduce an experiment whose results were drawn from /dev/urandom is pointless.

In fact I'm surprised the replication was so high. I was expecting it to be around 5% (i.e. results are totally random). So maybe there is some hidden merit to all of this

Re:No Paper Is That Good by nicoburns on Friday July 13, 2018 @08:27AM (#17522311)
> If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about.

This, 1000 times over. Unfortunately we seem to teach that statistical analysis is sufficient to infer causation in a lot our undergraduate science programs.

Re:No Paper Is That Good by traverseda on Friday July 13, 2018 @08:31AM (#17522338)
I wouldn't expect totally random because people have common sense. I'd expect it to be similar to the folk-wisdom of the time, since a lot of it is going to be repackaging of that same folk-wisdom, and negative results tend not to get published, or get massaged until they're positive.

I'd expect social-psychology to mirror the dominant opinions of people in other segments of the humanities, although it is of course hard to determine the direction of that relationship.

Re:No Paper Is That Good by UncleMeat on Friday July 13, 2018 @01:43PM (#17525022)
A serious problem is when people assume that because the replication crisis in psych is widely publicized that it is exclusive to psych and then they dismiss all of the findings in the field. In my experience the replication crisis is rarely used to criticize method but instead used to dismiss research out of hand, whether or not it is supported by other research that limits the likelihood of error.

I think that psych should be commended for attacking this problem. As a field they are funding replications, which basically never happens in any other field. We don't even know what the "replication crisis" would be in empirical CS. Given discussions with ML people I'd wager that the "replication crisis" in ML is at least as dramatic.

Science is messy. But this is why we should defer to experts who have vision over an entire field rather than focusing on individual papers as conclusive data points.

Re:No Paper Is That Good by spruciefic399 on Friday July 13, 2018 @06:46PM (#17527527)
To be consistent, you'd have to leverage your arguments against other fields with replication crises, which include neuroscience and oncology. It's really true of the biomedical sciences in general.

Psychology has a history of being sort of self-analytical (which makes sense if you're a field studying human behavior). Meta-analysis, for example, was largely incubated in psychology before it started expanding into other disciplines.

The criticism I'm aware of is really being leveled at the idea that this is somehow unique to psychology. It's like killing the messenger, or someone who is trying to fix the problem.