What a debate about p-values shows about science

Re:What a debate about p-values shows about science by kurthr on Monday October 02, 2017 @04:53PM (#15387866)

Many have discussed this, but like writing bad/buggy code, there are just too many (combinatorial) ways to get a negative result, and most of them are frankly boring, distracting, or worse than useless. You would need a review paper of all the most common failures, screwups, and anti-patterns to be particularly useful... preferably with a good write-up of proper procedure. Some labs publish good protocols, but many (academics!) consider them to be secrets to used for advantage over other labs.

Of course, based on the current p-hacking, publish at all costs environment, it's not clear you need to be particularly useful to be published. I like the idea that has been promoted of requiring a certain number of novel replications in a tenure application as one way forward... at least as a goal or professional shaming tool.

[ Parent | on HN ]

Re:What a debate about p-values shows about science by pjschlic on Monday October 02, 2017 @05:48PM (#15388249)

This sounds like a great case for publishing negative results: it will result in meta-analysis about what practices routinely cause bad results and then those papers will be used to reduce the number of bad results. Labs keeping secrets in how to do research seems like a bug not a feature.

That's just the externalities around publishing negative results. The major reason being that it will tell people where the minefields are, resulting in people not all reproducing the same negative results.

[ Parent | on HN ]

Re:What a debate about p-values shows about science by Sacho on Tuesday October 03, 2017 @02:36AM (#15390607)

Er, the type of the result(positive/negative) should be orthogonal to the methodology, from the point of the reviewer. Reviewers should be checking for all these many combinatorial ways to get an incorrect result, whether it's positive or negative.

It may be true that positive results are more useful than negative ones, but that doesn't mean we should put zero value in negative results. I think that's what parent was trying to point out.

[ Parent | on HN ]

Re:What a debate about p-values shows about science by danstanflan on Tuesday October 03, 2017 @07:29AM (#15391623)

Agreed, a negative result should not be confused with shoddily executed science. A solid methodology that produces a negative result could save other researchers from going down the same rabbit hole to come to the same conclusion. We should record these conclusions. Perhaps maybe part of the problem would be the increase in number of publishable results overloading editors and reviewers?

[ Parent | on HN ]

Re:What a debate about p-values shows about science by nzjrs on Tuesday October 03, 2017 @01:53AM (#15390458)

The code analogy is ok, but remember that the natural sciences are not engineering - a problem to be overcome.

There is great value in publishing negative results in the form "I hypothesized X, did Y, and turned out X was wrong". It's my experience these negative results are often well controlled and trustworthy as there are few things that make one trust a method as much as holding onto it for dear life because you have spent so much time building it. Incidentally that's not always as true for positive results where the incentives are tilted towards stopping once you get ahead.

[ Parent | on HN ]

Re:What a debate about p-values shows about science by admiralpumpkin on Monday October 02, 2017 @06:53PM (#15388671)

I agree, there are always more way to make mistakes, 2nd law and all, but the process of assessing good study design, good protocol, good statistical methods, etc. are all the same as for manuscripts with non-null results. Many of these issues will, in theory, have been sorted at the funding level.

What holds us back, it seems to me, isn't process, it's culture and inertia.

[ Parent | on HN ]