Re:What's the point of writing good scientific software?
by sophacles on Friday February 08, 2013 @03:45PM
(#5190022)
A large portion of my job, and my team's job, is programming for research at a university. We get "researcher code" that was used to write a paper, and turn it into something for the next set of researchers to build on. There are several other groups at this university that do the same thing.
I'm starting to think, there may be a field of study here. I regularly take software that "proves" some hypothesis or "shows some good results", tear it down, and throw some engineering at it, only to find that the results are not reproduced, or the benefits are severely diminished. Then I have to go track down the why... because until I can show it, it is assumed my fault.[1] The results of some of this are probably paper worthy themselves.
I think a useful field, or useful conference at least could be built for people in this typeof postions, studying the meta effects of software on research. How can we report the issues found, or the updates to the numbers, without putting black marks on the reputations of people who actually are doing good work?
Another interesting phenomenon that is worth study is that the refactoring/rewriting process often gets real results, but it turns out the mechanism for the improvement isn't what the original researcher thought/claimed. It is something perhaps related, perhaps a side effect, and so on. There needs to be a way to recognize both the original researcher, the programmer who found the issues, and the follow-up researchers who did some more determination of the problem.
[1] This isn't as antagonistic as it sounds. It is actually a nice check on my own mistakes. Did the differences in what the researcher did and what I did introduce some strange side effect? Did I remove a shortcut that wasn't actually a shortcut and I misunderstood? A hundred other things on both sides... Research has a large component of "we don't know what we're doing, axiomatically so", and as such it is a decent way of finding out more info.
I'm starting to think, there may be a field of study here. I regularly take software that "proves" some hypothesis or "shows some good results", tear it down, and throw some engineering at it, only to find that the results are not reproduced, or the benefits are severely diminished. Then I have to go track down the why... because until I can show it, it is assumed my fault.[1] The results of some of this are probably paper worthy themselves.
I think a useful field, or useful conference at least could be built for people in this typeof postions, studying the meta effects of software on research. How can we report the issues found, or the updates to the numbers, without putting black marks on the reputations of people who actually are doing good work?
Another interesting phenomenon that is worth study is that the refactoring/rewriting process often gets real results, but it turns out the mechanism for the improvement isn't what the original researcher thought/claimed. It is something perhaps related, perhaps a side effect, and so on. There needs to be a way to recognize both the original researcher, the programmer who found the issues, and the follow-up researchers who did some more determination of the problem.
[1] This isn't as antagonistic as it sounds. It is actually a nice check on my own mistakes. Did the differences in what the researcher did and what I did introduce some strange side effect? Did I remove a shortcut that wasn't actually a shortcut and I misunderstood? A hundred other things on both sides... Research has a large component of "we don't know what we're doing, axiomatically so", and as such it is a decent way of finding out more info.