Re:No One Peer-Reviews Scientific Software
by spamizbad on Monday November 30, 2009 @04:33PM
(#968735)
What's your view on the mountains of scientific code that depend on proprietary math kernels and in some cases proprietary compiler platforms?
As you may guess, I'm unhappy about it. In astronomy, almost everyone uses IDL, which is a proprietary, closed-source language. (It also sucks, but that's beside the point.) I think this is bad, because it locks practically the entire research field into a closed platform. (Though GDL is working to be a replacement. I don't know how conformant it is, though.)
Still, I think this is less of a problem than the high-level scientific code. Most of the low-level functionality is pretty simple and (hopefully) much better tested than most scientific codes.
Re:No One Peer-Reviews Scientific Software
by aristoxenus on Tuesday December 01, 2009 @12:47AM
(#969311)
I'd just throw in that if someone uses your code to try to reproduce what you've done, they may actually just be reproducing a bug -- or in the sinister case, reproducing a faked result. So a more general point in this whole discussion would be that making the code available isn't always as important as making the mathematics or other theory behind the code available, and "reproduction" becomes "reimplementation", or some other corroborating element of the process as described by timr a few comments down (http://news.ycombinator.com/item?id=968456).
(Of course for large software endeavors that isn't practical either. So mattheww's point about trust unfortunately resonates.)
As for the proprietary issue, I think sometimes it might actually help reproducibility: if you can just drop a few lines of script into Matlab and examine them, it's potentially a much smaller task with a lot more battle-worn, common-ground code factored out than examining or re-implementing a full library of open code.
And while I certainly don't expect any of my referees to go over the code line-by-line, one of the rationales I had for GPLing the code was to make it possible for someone to replicate my study. One of the basic tenets about publishing research is that articles should contain all the information needed to reproduce the results. This is not possible for simulation papers unless the source code is public (or maybe a binary is provided, but that has many pitfalls by itself).
Results generated with proprietary codes could just as well be hokus-pokus in my mind. It's not clear how it adds to the field unless others can reproduce and build on the results. And with reasonably complicated codes, it's not realistic to think that someone will implement their own version (though that is of course the absolute best thing).
So while I don't necessarily think scientific software should be peer reviewed, I'm very hesitant about results obtained with proprietary software.