News for hackers. Stuff that matters.
Re:Practices in source code sharing in astrophysics by Blahah on Friday April 26, 2013 @06:40AM (#5612256)
That's an extremely inefficient and costly way to validate a result! It would be rendered unnecessary in cases where just looking at the code invalidated the result.

I'm in academia (see my profile for details).

Re:Practices in source code sharing in astrophysics by J_Thomas on Friday April 26, 2013 @09:05AM (#5612648)
It's good and useful for independent researchers to collect their own data and analyse it. That's a check on the data collection methods.

It's also useful for others to check the analysis of trusted data. If they do it with their own methods and get a different result, then it's time to actually compare the methods.

It's a good thing for the original researchers to release their code too. But looking for bugs in the code is no substitute for doing independent analysis and independent data collection. It's an important supplement, but it isn't enough.

Re:Practices in source code sharing in astrophysics by Blahah on Friday April 26, 2013 @11:42AM (#5613524)
I agree with your points. The saving in the case of released source code is that, if the original analysis is flawed in such a way that there's no need to repeat it because it's wrong, then a large amount of expense and effort can be saved. Of course, if the original analysis holds up given the data and code, independent replication is a necessary thing.
Re:Practices in source code sharing in astrophysics by dalke on Friday April 26, 2013 @12:20PM (#5613712)
Sometimes validation can be done without knowing the actual algorithm or being able to rerun it.

For example, if a paper publishes a multiple sequence alignment of 100 sequences, then all you need to do is verify that it's at least as good as what other MSA programs generate. You don't need to be able to rerun the program.

Indeed, sometimes you can't. Someone could have manually aligned the sequence, and that process isn't reproducible.

You see this as well with fold prediction software, where it can be easy to show that a predicted fold is likely correct (ie, matches experimental observations), and where you don't necessarily care about the method used to get that fold prediction.

A genetic algorithm might be very sensitive to the compute environment. For example, the order of float operations generated by two different compiler settings, or by network traffic timings in a distributed system. This can lead to different minima, where the overall effectiveness is the same but the actual configuration is different. The GA search doesn't need to be reproducible; it's only the final effectiveness which is important.

For those cases, I don't see the need for access to the underlying software in order to validate the result.

Re:Practices in source code sharing in astrophysics by Blahah on Friday May 03, 2013 @07:33AM (#5649721)
The problem with the MSA question is that there is no objective measure of goodness. Most people compare them to a subjective 'gold standard', which is a manual alignment. Black box comparisons are not progressive: the only non-subjective advances can come from incorporating the results of sequence evolution experiments into the algorithms, and then the algorithm needs to be trusted, for which it needs to be open to scrutiny.

I agree about the genetic algorithm.