The thing is though, as I'm sure you're aware, the algorithm in abstract is utterly irrelevant. The algorithm as released is never actually executed, it is the implementation that gets executed.

One "trivial" detail in the implementation, one tiny little floating point rounding error, can throw the entire thing. We all know this. So why is the scientific community so unwilling to face this elephant in the room? And why is nobody else confronting them about it? Of course there's jgc, with his admirable record of confronting people about things, but that's the only example I can think of.

You would've thought the "Al Gore made up AGW to devalue my exxon shares" angst-brigade would've jumped on this, but they seem more concerned with digging through departmental email gossip.

"The algorithm as released is never actually executed, it is the implementation that gets executed."

If you use a different implementation of the algorithm, on the same data, and get a different result, you will find that there was an error. That's a lot easier than combing through someone else's code base.

A simple example: If Lab A says they took the square root of 16, you don't need their source code to know there's a problem, if their result is 5 and your result is 4. If you use their code, you might not notice that it's borked.

Still, this is not a good reason not to release code. We can deal with funny comments, bad jokes and dead code. Why not let people examine the code? They are not programmers who will be judged by the neatness of their code.

What if they don't have code, because they're using a commercial product to do the recording?

Well... That part has to be certified as a black box, but even if your code is a Mathematica notebook, it would be useful to have it published along the raw data.

Again, there is no reason to be embarrassed of comments like "this should never happen". My own code has parts that raise "ThisShouldNeverHappenError" when something that should never happen disregards my opinion and happens anyway. ;-)

This assumes errors by implementors are uncorrelated, which is almost certainly false. As programmers we often all make similar mistakes, particularly in exacting numerical code.

True, but the second implementation may use different, well-tested, mature code.

If the first lab used C++, and the second lab used Matlab and its libraries, and the third lab used trusted code based on NumPy, I would be surprised if their errors were strongly correlated. Brand new code? Sure. But not code they've used for several years through multiple experiments, or code that's part of a widely-used software package.

The thing is, labs working in similar areas most likely have ready, tested, trusted software toolkits for doing similar tasks, especially when it comes to data analysis.

It may seem to a programmer that the fastest, easiest thing to do would be to read the code, in fact the fastest, easiest thing to do is probably for another lab in the same field to do an independent analysis of the data with their own code.

"trusted code based on NumPy" <- (Just to note, a number of NumPy's routines have seemingly known numerical instability issues; I would not trust it, and was burned by it badly a few months ago. If you've used a specific function before and it seemed to work, then that's great; but if you haven't: watch out.)

You've implicitly accepted that the first implementation is correct, which I've found is one of the mental blocks that people defending this practice seem to have with frequency that surprises me. It isn't true. It isn't even the sort of thing you can try to debate your way out of, because it simply isn't true. Rather complicated numeric code written by non-software engineers is not something I'm willing to presume correct. I wouldn't presume it correct if it were written by software engineers, either, but at least then we'd probably have some sort of halfway reasonable testing to point at.

Further, the Kolmogorov complexity of a simulation is typically a wee bit larger than the Kolmogorov complexity of a square root implementation. That metaphor is beyond useless, it's deceptive.

"You've implicitly accepted that the first implementation is correct"

Not at all. The questions are 1) does the method described sound reasonable and 2) if I implement the same methods, with my toolset, and process the same data, do I get the same result?

If the results are different, using different implementations of the same methods, then it's possible that either party could have the error. But if the second party is reusing well-tested code of their own, or something from Matlab or whatever, then it is more likely that the unknown code of the original lab is the problem.

You might have a point about a simulation, but an awful lot of science isn't simulations, it's analyzing recorded data.

s/simulation/any real code/. It's still not a square root, which is a total straw man. If it were that simple, there'd be no code to write in the first place.

Okay, instead of 'square root' try spike sorting algorithms for recorded neuron activity. There are various ways of doing it, various software packages, free and commercial, ways of doing it in Matlab, etc.

(Ex: http://www.plexon.com/product/Offline_Sorter.html#Features)

Given a terabyte of recording data, I'd think it should be sufficient to specify the parameters they used for spike sorting. Someone else ought to be able to use a different package on the same data and obtain results similar enough to know if the first person fudged their paper.

Given a set of data, and a named algorithm, it ought to be possible to obtain the same result with any correct implementation of that algorithm.

That's begging the question. Yes, of course a correct implementation will by definition obtain the correct result. I'm questioning the casual way people assume they've got the correct algorithm implemented at all.

Is your paper nothing but off-the-shelf "spike sorting", or is that used as a component of something larger? Rolling back around to the original point of source code release being a desirable thing, if it's the latter, just knowing that this one library was used from what is probably literally a single sentence in the paper that reads "We used spike sorting software X to obtain sorted spikes", when presumably the library has knobs whose settings you don't know and you still don't even have their raw data to check the settings with, you still know virtually nothing about what was actually done. Source release ought to be standard.

The degree to which people who are putatively scientists will go to bat to defend making it difficult or impossible to replicate their experiments boggles my mind.

You're still obstinately missing the point in your nerd rage — letting other people run the same source code does nothing to replicate that part of the experiment — they need an independent implementation to do that! Releasing the source code publicly actively works against that goal because anyone who reads it won't be able to do a black-box implementation.

Besides, the code is almost always worthless compared to collected data. What would actually be most valuable and practicable is for groups to be running their proprietary code on each other's public data for confirmation.