I translated what you said in the context of mathematics:

“But the proof is secondary to theorem. The theorem and the discussion around how it was arrived at and what it means is the key thing. The proof is just there to show it’s true. You could write a proof for the same theorem ten different ways.”

Which is all true. But man it wastes so much time having to re-prove everything. Also some lemmas/theorems are so hard to prove. It’s much easier when you see some incredible statement and can’t believe it’s true to look at the proof and see where the mistake / contentious part is.

Ok but can you understand other fields are different to mathematics? In other fields we argue more informally.

Yes! There is "reproducibility" in the sense that you can run the code and get the same answer, and this can be very useful. But I don't see that as the core domain of the research paper, which is to describe a new discovery. The papers job should be to explain the discovery and give appropriate supporting evidence. This leads to a stronger form or reproducibility, like you say, which is "I understand the discovery and I can do it too". That's not the same as generating the exact result the authors' show. And a paper that is reproducible in the first sense but not in the second is of limited scientific value.

If anything, I'd like to see more focus on giving evidence of generality of a result, vs just sharing everything needed to get back the same specific result

Yes but chances are it only appears to work because the analysis code has bugs. So first i want to check the code and that it works before i put effort into understanding the idea.

So let me clarify.

You're saying that the majority of CS papers only appear to work because the analysis code has bugs?

And that checking the code (presumably also the analysis code) is easier than "understanding the idea"?

Neither of those ring true to me, but your mileage may vary.

Yes, a significant proportion of all scientific research is not reproducible e.g., "Over half of psychology studies fail reproducibility test" https://www.nature.com/articles/nature.2015.18248 Cancer research: "scientific findings were confirmed in only 6 (11%) cases." https://www.nature.com/articles/483531a

Those aren’t CS papers, social science and biology research have constraints that CS does not. I haven’t seen any evidence that there is anywhere close to that level of issue here. A couple of conferences adopted artifact review where an independent reviewer attempts to reproduce the experiments listed in the paper. Nearly all papers that participate do end up passing

One case where this happens a lot is papers that pick bad comparisons as state of the art. If you have the code, you can run it vs better configurations of existing tools to see if the promises still hold up.

> CS papers

I don't think any specific field was mentioned

Yes, that's what I tried to say. I can easily try a piece of code with my own data to verify that its results are plausible. I can't do that for a paper. So if I don't have the code, I might waste a significant amount of time trying to reproduce a fake paper.

Some papers are about an idea, but others (perhaps most others) are about results. And the results are very much dependent on the data and how you analyzed it.

The code is there to show that the idea is worth the discussion around.

Would the idea be valuable without the code? Yes.

If I've learned anything in my career it is that no, ideas are not valuable. There are vastly more bad ideas than good ideas. What makes an idea valuable is validation. Papers aren't to present ideas: papers are to present ideas that have been validated. We proposed an idea, we went and ran some experiments or gathered data some other way, and we concluded the idea was valid (or not valid). The point of this discussion here is that ideas that require huge amounts of computer effort to validate are prone to bugs. The conclusions cannot be relied upon to be validated without having the software available so that it, too, can be validated.

But there are so many valuable papers in CS that just presented an idea. If you ignored them you’d be ignorant of how to do 90% of modern engineering.

Likely the tech stack you use is built on a tower of ‘just idea’ papers.

Really? I've read a lot of SIGGraph papers, and sure, they didn't provide all the code. But you know that the code exists. And certainly for those there's a lot of trust. I think we're talking here about something different. Not "hey, you can use quaternions for animation" but "If you factor in hippopotamus usage, young adults experience 22% more flange per square doughnut, and we got all this data, and we ran it through 25 separate computer programs written in lolcode, and look, proof!"

> But there are so many valuable papers in CS that just presented an idea. If you ignored them you’d be ignorant of how to do 90% of modern engineering.

You are right. But that is why we are having this discussions, so we can improve situation.

Having even bad code (and corresponding data) available is always better than not. You can always just ignore it, and read the papers like today.

Honestly I am ok with just zip file of project directory that you have anyway, with hopefully list of versions of os, libs and programs used.

We could do a lot better than just a zip file, but that would be a nice start.

For some papers. Others make a claim about some statistically significant look at data that might not have any basis in reality because the code is wrong. A famous example being the R&R paper in economics where a second look at the showed massive mistakes in the excel document they were using, invalidating the central thesis. Unfortunately not before being used by the world bank for years as a metric for forcing austerity on countries.

https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Alleg...

Not if the code is wrong, and therefore the conclusion may be wrong. I'm no scientist, but I don't think the point of scientific papers is to get unfounded ideas out into the world.

I can list many major influential papers in computer science that described an idea and didn't really give any concrete code, where we're still using the idea today.

For example the paper on polymorphic inline caching, which is the key idea for the performance of many programming languages today, just described the idea, and didn't present any code. How was it evaluated? People sat and thought about it. Holds up today.

You can reason about an idea through other things than concrete code. Code is transient and incidental. Ideas persist.

I think you're talking past each other. Both are true under different circumstances. In some cases an abstract idea is the important takeaway. In other cases the central point of a paper is to present conclusions that were arrived at based on analysis of some dataset. If the code used to generate or analyze the dataset is wrong then conclusions based on it likely worthless.

A lot of times the idea is wrong (and thus not valuable), and that can’t be proven either way without the code and data. So an idea that depends on code without the code is less valuable.

We can only possibly gain from publishing the code, and lose by not publishing.

It's not like it takes a whole lot of time to just dump your code in a github repo once you're done and link it somewhere on the paper (if you wrote code at all while working on the paper).

Sometimes I did just want to run my own experiments with different datasets, and those algorithms aren't always trivial to implement :|

Facts and proofs are more interesting that unproven (and sometimes unprovable) ideas.

If nobody ever shared useful but unprovable ideas we'd worse off.

Throwaway because I have “skin in the game”…

Yep but we should still show we did actually simulate our idea, and the methodology that gave rise to the simulation. Not because of the code but to test at all a simulation we describe actually outputs what we propose

Not everyone is a programmer but they could find one to confirm, or better yet, invalidate code my team relied on