You would rather leave potentially incorrect work standing than have a bug corrected?
What the f*ck has happened to science.
This is fairly standard in the science industry (I use the term deliberately). I get rated (in nuclear and particle physics) on how many papers I've published, what indictions of recognition I get from my peers and at which conferences I've been invited to speak. Short of something like a Millennium Prize or a Nobel (one-in-a-million, and both still very political), there are almost no direct rewards to accuracy or importance.
Academics tend not to be the people with the most understanding or the best direct insight - they're the people with the most friends on committees and who do the most fashionable things (usually badly). And so they appoint new academics who are a little bit worse than them (no point in opening yourself to competition - instead bring in people who are going to be grateful to you!) and the cycle continues.
The most sickening part of it all is that who are the ones who perpetuate this system? It is us (other academics).
What can we do to change it? I love research. Having almost 10 years of experience but still earning less than if I had gone straight into industry after a masters degree wears you down though.
I think ultimately it is because scientists add little of value to society in the short term. An MD who has a comparable number of years of education can expect to earn 10X that of an equivalent scientist because they value immediately.
I like Feynman's quote "It's a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty--a kind of leaning over backwards. For example, if you're doing an experiment, you should report everything that you think might make it invalid--not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you've eliminated by some other experiment, and how they worked--to make sure the other fellow can tell they have been eliminated." [1]
> It sounds like you don't want to release your code because someone else might find a bug that you missed
I already said at the end of my post that I decided to release my code. Everyone will be able to see it.
> You would rather leave potentially incorrect work standing than have a bug corrected?
Of course not. You missed the whole point of my post. My point was "Here's a situation in science that needs fixing". "People are hesitant to fix it because...". "I'm going to personally work towards a solution."
Let me ask: what incentive does any scientist have at all to publish their code? You're not going to make money off of it. It's in an obscure niche so you're not going to be world-famous with it. You may get citations to your work, but you're just making yourself vulnerable to having your reputation destroyed because of a bug that nullified all of your articles' results. This is why nobody wants to do it. I'm not saying it's right; I'm saying that it's the status quo.
Most big scientific packages are funded by the DOE, NSF, and others. That's likely the only reason they are even out there.
holy moly x 2
How about that it represents a fuller account of what you did and how you did it? (bugs or no bugs). Isn't scientific publishing supposed to be about reporting what you did as accurately as possible so that others can (1) understand and (2) replicate?
BTW the * in my f*ck from above stands for "ra", what did you think it stood for?
> How about that it represents a fuller account of what you did and how you did it?
So what? Journals never require your code to be submitted. It's not going to increase your article's chance of acceptance. And nobody asks for your code anyway. Why should I publish it if it's not going to bring me any benefits?
> Isn't scientific publishing supposed to be about reporting what you did as accurately as possible so that others can (1) understand and (2) replicate?
In an idealized world, yes. But nobody else does it so why should I?
From the perspective of what "science" is claimed to mean, namely the advancement of human knowledge in a way which is repeatable and verifiable, it seems axiomatic that sharing your algorithms, code, and data are necessary and beneficial to the scientific community.
As a researcher, you don't gain a lot from publishing YOUR code ... but you sure might gain a lot from being able to re-use someone else's code in your domain, or more easily replicate someone else's experiment.
In short: You should share your code because it's the Right Thing to Do if you want to grow human knowledge.
It seems that way, but it isn't. If a cultural foolishness causes you to lose significant credibility undeservedly, it's actually more beneficial to withhold things that could damage it so that you can continue your essentially still-very-useful work.
You could spearhead the fight against said cultural foolishness, but that's time and energy spent doing something other than the work you want to do and are best at.
Someone has to, but why you? This is the Bystander Effect.
So, I guess I can see that argument here. Just because the software has a "bug" isn't sufficient to conclude the results are not accurate. The prejudicial value outweighs the probative value.
wow. has it come to this? really? maybe you should reconsider your career choice
Science shouldn't be like this but I think it's zero sum.
You might want to automate the process of re-running your analysis, though. It's a good idea in general, and especially if you anticipate needing to make minor tweaks to the software involved.
In contrast imagine an academic scenario where I'm testing a hypothesis using someone else's software. How do I notice a bug? I'll notice the bugs where the output is in the wrong format for example. More subtle bugs I won't notice because I don't have any expectation on the results. This will affect the conclusion I draw though.
I recently introduced my advisor to Github, and he thought it was a good idea; however, there were a few hesitations. The first, most importantly, is the likeliness of a bug. If you put your code on a very public website like Github, there is a chance it's going to be scrutinized by everyone in your field.
Now, unless you are one of the best programmers who has ever lived, there are bound to be bugs in your software, and when someone discovers them, it could have a deleterious effect on any journal articles you've written that used that code. The issue is that even though most bugs do not lead to significant changes in results, you would still need to redo all of your data to make sure that is the case. The software industry has long recognized buggy software as a reality, but I don't think the scientific community is as tolerant of it (hence the reason a lot of people hide their code).
For my MD simulations, I use the well-known LAMMPS package. Bugs in it are discovered all the time! (http://lammps.sandia.gov/bug.html). So I think there needs to be a collective realization among the scientific community that these are bound to occur and authors of journal articles can't be persecuted all the time for it. A lot of computational work is the art of approximation so I would just lump "human incompetency" under one of those approximation factors.
Despite this risk, I think I'm still going to release my code at some point as I would personally welcome critique and improvement suggestions. I'd like to think I'm a better coder than most scientists since I've been coding since I was twelve in multiple language paradigms and have won a major hackathon, but eh, who knows. I'm quite sure my environment isn't up to industry standards because I've always coded solo rather than in a team.