News for hackers. Stuff that matters.
Re:Neutrino experiment affirms faster-than-light claim by jerf on Friday November 18, 2011 @11:04AM (#3252141)
You've implicitly accepted that the first implementation is correct, which I've found is one of the mental blocks that people defending this practice seem to have with frequency that surprises me. It isn't true. It isn't even the sort of thing you can try to debate your way out of, because it simply isn't true. Rather complicated numeric code written by non-software engineers is not something I'm willing to presume correct. I wouldn't presume it correct if it were written by software engineers, either, but at least then we'd probably have some sort of halfway reasonable testing to point at.

Further, the Kolmogorov complexity of a simulation is typically a wee bit larger than the Kolmogorov complexity of a square root implementation. That metaphor is beyond useless, it's deceptive.

Re:Neutrino experiment affirms faster-than-light claim by jonhendry on Friday November 18, 2011 @11:22AM (#3252229)
"You've implicitly accepted that the first implementation is correct"

Not at all. The questions are 1) does the method described sound reasonable and 2) if I implement the same methods, with my toolset, and process the same data, do I get the same result?

If the results are different, using different implementations of the same methods, then it's possible that either party could have the error. But if the second party is reusing well-tested code of their own, or something from Matlab or whatever, then it is more likely that the unknown code of the original lab is the problem.

You might have a point about a simulation, but an awful lot of science isn't simulations, it's analyzing recorded data.

Re:Neutrino experiment affirms faster-than-light claim by jerf on Friday November 18, 2011 @01:58PM (#3252884)
s/simulation/any real code/. It's still not a square root, which is a total straw man. If it were that simple, there'd be no code to write in the first place.
Re:Neutrino experiment affirms faster-than-light claim by jonhendry on Friday November 18, 2011 @05:35PM (#3253788)
Okay, instead of 'square root' try spike sorting algorithms for recorded neuron activity. There are various ways of doing it, various software packages, free and commercial, ways of doing it in Matlab, etc.

(Ex: http://www.plexon.com/product/Offline_Sorter.html#Features)

Given a terabyte of recording data, I'd think it should be sufficient to specify the parameters they used for spike sorting. Someone else ought to be able to use a different package on the same data and obtain results similar enough to know if the first person fudged their paper.

Given a set of data, and a named algorithm, it ought to be possible to obtain the same result with any correct implementation of that algorithm.

Re:Neutrino experiment affirms faster-than-light claim by jerf on Friday November 18, 2011 @09:26PM (#3254372)
That's begging the question. Yes, of course a correct implementation will by definition obtain the correct result. I'm questioning the casual way people assume they've got the correct algorithm implemented at all.

Is your paper nothing but off-the-shelf "spike sorting", or is that used as a component of something larger? Rolling back around to the original point of source code release being a desirable thing, if it's the latter, just knowing that this one library was used from what is probably literally a single sentence in the paper that reads "We used spike sorting software X to obtain sorted spikes", when presumably the library has knobs whose settings you don't know and you still don't even have their raw data to check the settings with, you still know virtually nothing about what was actually done. Source release ought to be standard.

The degree to which people who are putatively scientists will go to bat to defend making it difficult or impossible to replicate their experiments boggles my mind.

Re:Neutrino experiment affirms faster-than-light claim by blasdel on Saturday November 19, 2011 @12:03AM (#3254644)
You're still obstinately missing the point in your nerd rage — letting other people run the same source code does nothing to replicate that part of the experiment — they need an independent implementation to do that! Releasing the source code publicly actively works against that goal because anyone who reads it won't be able to do a black-box implementation.

Besides, the code is almost always worthless compared to collected data. What would actually be most valuable and practicable is for groups to be running their proprietary code on each other's public data for confirmation.