Re:Cancer reproducibility effort faces backlash
by logicallee on Saturday June 27, 2015 @03:13PM
(#9791221)
Isn't this actually an attractive ethical hazard† (in a very broad incentive sense) - and as such, to counteract it couldn't we actually encode an ethical obligation to immediately publish the data from any experiment, to counteract this hazard? Just in any old place, not as a full paper.
You could re-run your experiment of course if you thought there was some experimental methodological error, but as you disclose somewhere your first, your second, and your third dataset all showing the same data with more or less the same methodology, you would have to show increasing confidence in your fourth or fifth dataset (the one you would otherwise publish alone), because it has to explain all of the earlier datasets as statistical flukes only: you would no longer have the incentive to run the fourth experiment alone and publish it without reference to earlier trials.
To give an example, suppose anyone is rewarded by publicity if they show that flipping U.S. coins favors heads by more than 2%. This is temptingly easy to do if you don't publish experiments that don't statistically prove that: you just keep re-running the experiment until you get the result you want at the p-value you want, so if the reward for the result is more than the number of experiments you need to do to get it times the cost of each experiment (which can be truly tiny), the experiment presents an attractive ethical hazard.
But if you are ethically forced to publish (or even summarize) your earlier complete experiment and dataset really in any old place, then if there is no actual difference it becomes vanishingly unlikely that you can suddenly prove your theory and explain all of the earlier experiments as statistical outliers. You would stop exploring after your second or third experiment. (Which you would still quickly summarize.) This does however present an added burden to researchers, especially if they quickly test something before really refining the methodology to do so. So these quick disclosures could still be considered quite dirty and not very meaningful. However their disclosure would give a good indication regarding how strong a result really is. (i.e. by glancing at how much dirty data precedes the actual experiment being published.)
-
† I'd like to recall the words "moral hazard" because it does indicate that people are tempted toward the bad behavior. But economically I think that term is too specific (meaning risk-taking where someone else has to clean up in case of failure) -https://en.wikipedia.org/wiki/Moral_hazard
You could re-run your experiment of course if you thought there was some experimental methodological error, but as you disclose somewhere your first, your second, and your third dataset all showing the same data with more or less the same methodology, you would have to show increasing confidence in your fourth or fifth dataset (the one you would otherwise publish alone), because it has to explain all of the earlier datasets as statistical flukes only: you would no longer have the incentive to run the fourth experiment alone and publish it without reference to earlier trials.
To give an example, suppose anyone is rewarded by publicity if they show that flipping U.S. coins favors heads by more than 2%. This is temptingly easy to do if you don't publish experiments that don't statistically prove that: you just keep re-running the experiment until you get the result you want at the p-value you want, so if the reward for the result is more than the number of experiments you need to do to get it times the cost of each experiment (which can be truly tiny), the experiment presents an attractive ethical hazard.
But if you are ethically forced to publish (or even summarize) your earlier complete experiment and dataset really in any old place, then if there is no actual difference it becomes vanishingly unlikely that you can suddenly prove your theory and explain all of the earlier experiments as statistical outliers. You would stop exploring after your second or third experiment. (Which you would still quickly summarize.) This does however present an added burden to researchers, especially if they quickly test something before really refining the methodology to do so. So these quick disclosures could still be considered quite dirty and not very meaningful. However their disclosure would give a good indication regarding how strong a result really is. (i.e. by glancing at how much dirty data precedes the actual experiment being published.)
-
† I'd like to recall the words "moral hazard" because it does indicate that people are tempted toward the bad behavior. But economically I think that term is too specific (meaning risk-taking where someone else has to clean up in case of failure) -https://en.wikipedia.org/wiki/Moral_hazard