Well yes, but no. You want to avoid any systematic bias, which if you reuse tooling/instruments/code, you run the risk of, but source code is also something which can be peer-reviewed.

Reproducing the code base from scratch just isn't going to be tractable for large scientific pieces of software (e.g. CERN).

Better to have the source code and dig through it carefully as part of the review and replication than insist everyone write their own cleanroom copy just to avoid systematic bias.

Suppose someone decided to build another LHC, but to save money they would use exactly the same design, constriction crew, and code used to build the first one. Would you consider that a perfectly reasonable choice or would it seem risky?

That said I am all for peer reviewing code, but that’s not where this ends up. People are going to reuse code they don’t bother to dig into because people are people. The obvious next step would be to reject any paper that’s reusing code from a different team, but that’s excessive.

That said reusing code or data is much safer than reusing code and data.

I’m going to be upfront and say I don’t understand your position You seem to be making a number of questionable assumptions: 1) That people would blindly reuse code to the harm of science, including on a multi-billion dollar project like an LHC. That’s not likely to ever happen. 2) That rejecting those who plagiarize others’ code in a peer-review process is somehow excessive or problematic.

I can’t begin to understand where you are coming from.

I'm not sure if I have exactly the same concerns as the person you're replying to, but I've definitely noticed problems coming from this form of "replication" through just re-running existing code. I don't think that means code shouldn't be released, but I think we should be wary of scientific code being reused verbatim too frequently.

If existing code is buggy or doesn't do what its authors think it does, this can be a big problem. Even if the idea is correct, whole swathes of downstream literature can get contaminated if authors didn't even attempt to implement an algorithm independently, but just blindly reused an existing docker image that they didn't understand. I consider that poor practice. If you claim that your work is based on Paper X, but what you really mean is that you just ran some Docker image associated with Paper X and trusted that it does what it says it does, instead of independently reimplementing the algorithm, that is not replication.

In older areas of science the norm is to try to replicate a paper from the published literature without reuse of "apparatus". For example, in chemistry you would try to replicate a paper using your own lab's glassware, different suppliers for chemicals, different technicians, etc. Analogies here are imperfect, but I would consider downloading and re-running an existing docker image to be dissimilar from that. It's more like re-running the experiment in the first lab rather than replicating the published result in a new lab. As a result it can, like re-running a chemistry experiment in the same lab, miss many possible confounds.

Of course, you do have to stop somewhere. Chemistry labs don't normally blow their own glass, and it's possible that the dominant way of making glass in your era turns out to be a confound (this kind of thing really does happen sometimes!). But imo, on the code side of things, "download and rerun a docker image" is too far towards the side of not even trying to replicate things independently.

For that reason a special multimedia sharing tool was created, which is now called WWW. So the international physicist community could share their code, papers and CAD designs for the LHC experiments. Quite a success for them, but academia is still resistant

It's not reasonable, but for different reasons. Building a carbon copy of LHC would not add anything new in the way of science. A better example would be LIGO. Adding another gravity wave detector of the exact same spec but halfway around the world would be fantastic, because it increases our triangulation ability, and keeping the engineering, hardware, and software the same reduces cognitive load of the scientists running it. Yes that means any "bug" in the first is present in the second, but that also means you have a common baseline. In fact there will inevitably be discrepancies in implementation (no two engineering projects are identical, even with the same blueprint), and you can leverage that high degree of similarity to reduce the search space (so long as subsystems are sufficiently modular, and the software is a direct copy).

The original comment was with respect to some n-back training program. There's so many other potential places of bias in an experiment like that, that you'd be foolish not to start with the exact same program. If an independent team uses a different software stack, and can't replicate, was it the different procedure, software, subjects, or noise?

The first step in scientific replication is almost always, "can the experiment be replicated, or was it a fluke?" In this stage, you want to minimize any free variables.

It's a matter of (leaky) abstractions. If I'm running a chemistry replication, I don't need the exact same round bottom flask as the original experiment; RBFs are fungible. In fact I could probably scale to a different size RBF. However, depending on the chemistry involved, I probably don't want to run the reaction in a cylindrical reactor, at least not without running in the RBF first. That has a different heating profile, which could generate different impurities.

Likewise, I probably don't need the exact same make/model of chromatograph. However, I do want to use the same procedure for running the chromatograph.

If it doesn't replicate on the same equipment with the same code, then there's no reason to worry about the rest.

Ideally, that would be a concern of the peer review. When I finished my undergraduate degree, I had to present a paper describing a program. I had to show that program working as part of my presentation. Anyone reading my "paper" and knowing I got a passing grade already knows the code I supplied works with the data I supplied, so I don't think it's that important for them to run it themselves.

Essentially, this would be like trying to reproduce a paper and starting by checking if their mathematical apparatus is correct. It's not useless, and it can help detect fraud or just plain bad peer review of course, but I wouldn't call that an attempt to reproduce their results per se.

The only person who really understood how to do independent validation. Of course the code MUST be different.

Why is it not a good idea to first redo with the original code?

Also doing an independent implementation doesn't mean you should start there - what if it doesn't replicate with the same code?

It could be a nice quick sanity check, in the sense that if they've completely lied or you've completely misunderstood how to use the program, you won't get the same results. So it could tell you that you shouldn't even bother trying to replicate their claims. But there's a risk that people might mistake re-running the code for reproducing the findings of the paper.

The paper is entered into the scientific record, not the code, which will inevitably become obsolete (old language, old frameworks, implementation details tied to old hardware or operating systems, maybe some source code will be lost, github will go out of business). If the code is necessary, then crucial details have been left out of the paper, so it is not reproducible (although there are some journals that let you submit code as an artifact).

Pseudocode is better, IMO.

The risk here is that the original code contains a bug that is inadvertently reproduced by the replicating scientists after reading it. It can easily happen in some fields that some computation looks very plausible in code, but is actually incorrect.