News for hackers. Stuff that matters.
Re:About 40% of economics experiments fail replication surve... by zamfi on Monday September 17, 2018 @04:17AM (#18004058)
Well, for one, because the standard for publication in many fields is “there’s a less-than-5% chance that we observed these effects because of coincidence”.

Combine that with people not publishing negative-results studies, and all it takes is someone doing 20 studies (or worse, 20 analyses on the same data set) in order to find that 1-in-20 chance occurring...by chance.

Of course, this has little to do with research itself and much more to do with the standards researchers hold each other to.

Re:About 40% of economics experiments fail replication surve... by viraptor on Monday September 17, 2018 @05:55AM (#18004481)
While you're right about the 5%-or-less chance, it's still a 5% chance we're aiming for. If we end up with 40% over a large number of papers, something went wrong.

Not publishing the negative results shouldn't affect this number. They're not included in the 5% chance in the first place.

Re:About 40% of economics experiments fail replication surve... by evandijk70 on Monday September 17, 2018 @06:23AM (#18004605)
This is a common misunderstanding of p-values.

There is a 5% chance of observing the effect, if the effect is not there.

I think the difference is best illustrated by an example:

If you have 20 researchers investigating a hypothesis that turns out to be false 1 of them will report evidence for a false hypothesis, which will likely not replicate. Thus, if for every true hypotheses investigated, 20 false hypotheses are investigated, and the 19 researchers with false hypotheses do not report their result, that means that 1/2 of the reported positive results will not replicate.

Re:About 40% of economics experiments fail replication surve... by coldtea on Monday September 17, 2018 @01:00PM (#18007690)
That's a cyclical argument though to what the parent asked though.

"Research is OK to have ac40% non-replication rate" because "the standard for publication in many fields is low"