News for hackers. Stuff that matters.
Re:Eleven strategies for making reproducible research the norm by somenameforme on Saturday November 25, 2023 @05:29AM (#38412616)
You're also leaving out the biggest issue. Journals generally don't want to produce negative results. If you spend researching [shocking possibility] and it turns out that [shocking possibility] isn't true, you're not getting published. It motivates everything from HARKing [1] to outright data manipulation. By contrast if negative results were seen as valuable, then none of this is an issue.

On the other hand, it really is the case that there's just not much of any value in learning that [shocking possibility] is, as everybody would naturally expect, indeed not the case. And filling up limited journal space with such discoveries would seem to be counter-productive, at best. And when you have limited space/funding for researchers, one guy who keeps proving everything everybody knows to be false, to be false, is always going to be perceived as less valuable than one making [shocking discovery] [... which ends up being proven false years later].

[1] - https://en.wikipedia.org/wiki/HARKing

Re:Eleven strategies for making reproducible research the norm by edgyquant on Saturday November 25, 2023 @08:22AM (#38413317)
So just make sure someone unaffiliated has to be able to reproduce whatever research has been conducted. Tough luck if it doesn’t make it, that’s why you do your best to ensure you’ve verified it’s a real result.
Re:Eleven strategies for making reproducible research the norm by staunton on Saturday November 25, 2023 @05:52AM (#38412702)
By "journal space" you of course mean journal prestige that isn't unlimited. The point of science journals is gatekeeping.
Re:Eleven strategies for making reproducible research the norm by QuesnayJr on Saturday November 25, 2023 @07:55AM (#38413198)
It can also be hard to judge whether replication failed because the result is bogus or replication failed because the replication team is themselves incompetent.
Re:Eleven strategies for making reproducible research the norm by edgyquant on Saturday November 25, 2023 @08:25AM (#38413333)
Did they follow the exact same steps claimed to result in something? Did it result in that thing?

The repetition team being incompetent sounds like a cop out. The researcher did a bad job and it’s on them to explain better etc in that case. No excuses, if it can’t be reproduced it isn’t taken seriously no exceptions

Re:Eleven strategies for making reproducible research the norm by bluGill on Saturday November 25, 2023 @08:43AM (#38413442)
How do you know which? Both will point fingers at the other.
Re:Eleven strategies for making reproducible research the norm by QuesnayJr on Saturday November 25, 2023 @09:41AM (#38413810)
You have two groups of people. Either is equally likely to be incompetent.
Re:Eleven strategies for making reproducible research the norm by bluGill on Saturday November 25, 2023 @08:42AM (#38413437)
The biggest problem is honestly obtained incorrect results. If you run 1000 experiments across 1000 labs. Few will statistically not notice a mistake and get a wrong result. That wrong result is then published as it is surprising.
Re:Eleven strategies for making reproducible research the norm by somenameforme on Saturday November 25, 2023 @10:56AM (#38414271)
I think there are some strong arguments against this. The first is numerical. Fields like social psychology are seeing replication rates as low as the twenties. And not just from low hanging fruit from but from journals like "The Journal of Personality and Social Psychology", which has one of the highest impact factors across all psychology journals, and a 23% replication success rate! [1] This [2] is a Google search for site:nytimes.com "Journal of Personality and Social Psychology". It's interesting seeing how many [shocking discovery]s, many which end up being shared on this site, come from this particular journal.

Furthermore, I think you can often see poorly done science in the papers themselves. They will use suggestive wording in surveys, unreliable sources for sampling such as Amazon Mechanical Turk, and maybe one of the biggest tells is measuring a large number of unnecessary variables. That does very little to further your experiment, but absolutely ensures you can p-hack your way to a statistically significant result. Another is ignoring such patently obvious viable confounding issues, that one can't reasonably appeal to Hanlon's razor.

[1] - https://en.wikipedia.org/wiki/Replication_crisis#In_psycholo...

[2] - https://www.google.com/search?q=site%3Anytimes.com%20%22Jour...

Re:Eleven strategies for making reproducible research the norm by naasking on Saturday November 25, 2023 @10:56AM (#38414272)
> If you spend researching [shocking possibility] and it turns out that [shocking possibility] isn't true, you're not getting published

But this simply isn't true in physics where negative results are very common. This is at least an existence proof that this can work, people just have to get their heads straight on what research means.