This mirrors my experience where a lot of the post-docs carried war stories of things like lab/country-specific humidity playing a role in synthetic methods succeeding (or failing). There were a lot of dark arts/tricks of the trade that people carried around with them: stuff like going that extra mile to dry things of water super thoroughly (even if it was not mentioned in the paper we were referencing).
An approximate CS analog: Writing great commit messages, and using an SCM.
You don't have to do them, nobody writes it up because those who know regard it as trivial, but if you don't do them, almost nothing works. Everyone who knows what they're doing does it.
This is definitely not the analogue. The analogue is always doing a clean install of the OS before running your experiment. Or only ever using Arch version x.y.z for replicating lab 1 and maybe a.b.c for replicating lab 2.
It's knowing all the magic undocumented JVM flags ahead of running the application.
Some you know to use as part of war scars/best practices. Some are just pure inside information from working in that lab or having a personal relationship with people in that lab.
However, for some I know, it's really difficult to get in, because there are many more journals of positive than negative results, and there are probably many more negative results than positive... so your negative result had better be especially interesting.
I just wish there were more, or at least more journals rewarding sound methodology instead of unsound "results". Who knows how much effort is wasted reproducing sound, but unremarkable studies simply because they weren't published
Obviously there would be no awards for the poor outcomes and failed results and we would be back to square one where no one is incentivized to look at them positively.
This is of course some work, but if you really want it to exists it’s better not to wait for someone else to do it. Also allowing anonymous authors would help I think (to avoid the name being associated with failure).
[1] https://www.openedition.org/15974?file=1
Edit: openedition.org is for humanities, but there probably is alternative for other fields.
Creating a journal is a tremendous task, and only highly respected researchers have any chance.
> if you really want it to exists it’s better not to wait for someone else to do it
Again, most people simply cannot create a credible journal. Let's be realistic here.
Example: https://www.princeton.edu/~joha/Johannes_Haushofer_CV_of_Fai...
Very unlikely to be effective.
Time and again people think that when they have a problem it must mean the punishment was not severe enough. It won't work.
The problem is that good science is not rewarded enough. Nobody rewards you for having published reliable stuff five years ago.
I think a few things could improve the quality and discovery of published papers:
-After so many publications, could it be mandatory for random samples in that author’s publication be tested? I get that there is a limit on resources, but some advanced undergrads could do this with guidance.
-I would love to see some version of a journal of failures. That is, well intentioned research that had poor outcomes. It was so frequently in Chemistry research that my compounds were useless or the methods to synthesize them did not work, and it would be helpful to document that. Unfortunately, there is no “Journal of Failed Chemistry.” Only research that ostensibly make a contribution with a clear outcome get published. So much time wasted experimenting where you could save another scientist time and encourage them down another path.