Re:Cancer reproducibility effort faces backlash
by zzleeper on Saturday June 27, 2015 @03:19PM
(#9791235)
I see a very similar mechanism in economics.
Many fields (macroeconomics, IO, etc.) write models that end up with some type of calibration which is implemented in thousands of lines of code. Those lines are written by RAs with almost no programming experience, so what happens is this:
While (result!=expectation)
Ask assistant to look for bugs and repeat simulation
The end result is that you don't stop when there are no more bugs, you stop when you've got the coefficient signs you want, and then get published.
Only if your paper has a high impact, has the code and number open, AND was coded in a very easy language, people discover the flaws:
"the paper ... was, and is, surely the most influential economic analysis of recent years [...] First, they omitted some data; second, they used unusual and highly ques ionable statistical procedures; and finally, yes, they made an Excel coding error"
Many fields (macroeconomics, IO, etc.) write models that end up with some type of calibration which is implemented in thousands of lines of code. Those lines are written by RAs with almost no programming experience, so what happens is this:
While (result!=expectation) Ask assistant to look for bugs and repeat simulation
The end result is that you don't stop when there are no more bugs, you stop when you've got the coefficient signs you want, and then get published.
Only if your paper has a high impact, has the code and number open, AND was coded in a very easy language, people discover the flaws:
"the paper ... was, and is, surely the most influential economic analysis of recent years [...] First, they omitted some data; second, they used unusual and highly ques ionable statistical procedures; and finally, yes, they made an Excel coding error"
http://www.nytimes.com/2013/04/19/opinion/krugman-the-excel-...