Peronsally, IMO: you should submit methodology--including the data analysis and statistical tests you'll perform, to a journal before the study has been conducted. Then, they say yes or no and if yes you perform the study and as long as it is performed to the approved specifications, you get published.

It sounds nice, but this will make for very boring journals with 90% negative results due to problems with the methodology.

Here an example of a typical project, where researchers are interested in the role of gene X in blood vessel formation in cancer, which could make it a good target for a cancer treatment. They want to knock the gene out and measure the blood vessel formation in an in vitro model.

Publication 1.

- We attempted to knock this gene out in this cell line, but due to an infection the cell line died. The installation was sterilized, the stored master cell bank was unfrozen, and we updated our sterilization protocols.

Publication 2.

- We attempted to knock this gene out in this cell line, but sequencing showed that there were off-target effects introduced. Reagents should be changed to be more specific.

Publication 3.

- The gene was succesfully knocked, but we found that a homologous gene appeared to activate the same pathway. The homologous gene should be knocked out as well to study the function of the gene.

Publication 4.

- The gene was successfully knocked out by disabling the gene and its homologue. The assay we used to study blood vessel formation was flawed. Due to insufficient oxygen supply, the positive and negative controls for blood vessel formation did not grow. New experiments should increase oxygen flow.

Publication 5.

- We successfully studied the function of the gene and found that it does not impact blood vessel formation in cell line x.

This is how the majority of research projects end up.

However, other researchers will be looking for a biological target to treat cancer X, and the information obtained by this series of experiments is not interesting. This is just one out of 99% of 30,000 genes that are not suitable treatment targets.

The conclusions are only interesting for researchers focused on gene x (and maybe its homologue), and potentially for researchers developing methodology. Questions answered were different than the original ones posed.

Wouldn't that just put the journals in control of what studies get performed (as well as what gets published)? Seems weird to give all that power to the journals.

Well, they are already in control of what studies get published, what isn't much different.

I do think they shouldn't control that, but it's a deeper change, for a different gain.

It is a lot different because if the study isn't published in one journal, it could always be published in another. In this situation, the study wouldn't be done at all until a journal agreed to publish it.

You could submit your proposal to other journals as well though

If I was a journal, and I was promising to publish a study once it is completed, I'd ask for exclusivity rights.

Which goes to show the whole issue with this idea.

Again that didn't change anything when compared to the current situation.

Journals do ask for exclusivity when they publish things. The don't ask for it just for submission. And the same reasons for why they don't now (nobody would submit anything to them) apply for the OP's idea.

In fact, all the changes only make the life of the researcher easier, because he would only have to present a proposal for the journals to deny. Now he has to complete his research before he gets the denial. (It also makes the life of the journals much harder, for the same reason.)

> Again that didn't change anything when compared to the current situation. Journals do ask for exclusivity when they publish things. The don't ask for it just for submission.

You contradict yourself. It absolutely changes things.

> Now he has to complete his research before he gets the denial.

It could degrade the incentive for the researcher to do great work, if they knew that they would be published.

There are already pre-print servers where you can pre-register your studies. It is not mandatory, but it is a think.

Despite his dubious past, the billionaire John Arnold has focused time and money to amend scientific research. I've read maybe 2 years ago about that endeavor and followed him on Twitter ; I found it quite interesting and original. For example, he offers researchers to save their initial data (before any screening) so that other researchers can check which cases were thrown out of the study. He also funded reproduction tests. I haven't followed since, so I hope he kept on doing good work.

This is my first time learning about him and haven't been exposed to much more than the wikipedia article - what about his past is dubious?

> It shouldn't fall to every individual researcher to be fully... honest.

Really? I don't know, "Don't be a fraud" might be a good Rule No. 1

Fraud is only a small part of the problem. No open data, poorly described methodology so reproductions sometimes aren't even possible, too little funding for reproductions, p-hacking, no incentives to publish negative results, journals incentivized to publish sensational results to increase their impact factor even though sensational results are likely false thus perpetuating misinformation. These are the main drivers.

It’s more that a system should not allow a single fraud to shape its understanding and its policy. Assume fraud exists.

Or even requirements for replication amongst groups with separate funding sources.

If everyone needs replication for publishing, the incentive to replicate other’s work increases.

In addition there needs to be tiers of publication that are purely objective. Maybe something along the lines of this paper was replicated by 3 different teams and the results hold up. Like a replication score or count- this would be desirable since then students could apply for funding to replicate papers.

The problem is that for any sufficiently complex or advanced research there are enough variables to contend with that there are 1000s of ways to get "no repeatable result" and only a few to actually replicate. Knowing which variables need to be controlled is part of the expertise, training, and lore. Knowing when those variables are getting out of control and how to fix them early (knowing which knob or often combination of knobs are required) is yet more expertise and experience. This is true in physics, chemistry, biology, and electronic design. Getting a null result really is often simply a failure to properly run the experiment... and as a scientist it would be very easy to follow any documented "recipe" and fail to get the proper result in hundreds of different ways. That does not make one a great scientist. You can't even tell me every way it could be screwed up. I can always come up with a new one!

Programming is such a wonderfully controlled environment, it's digital, replicable, and _almost_ math-like. You can know the variables. You can trivially replicate a program, or dig down into the OS... but every physical experiment out there has much more common and worse problems than Spectre or Rowhammer waiting to destroy it. It's the cosmic rays.

Your equipment/materials/specimens gets contaminated or reacts or ages. Your ultrapure water isn't. There's a vibration that changes everything. Your calibration sample changes. Cleaning staff used a new cleaner. The AC turned on. Build your experiment to be impervious to all of these and everything else I can't think of... or use the intelligence of an expert. Get there faster, cheaper, and find better ways of doing it along the way.

You'ld be right to say that it should all be documented, but the problem is that you basically need to be an expert to even understand the variables and to those experts many variables are so obvious to control they wouldn't think of doing otherwise. To give concrete and extremely valuable examples in manufacturing: EUV systems for lithography, OLED deposition and patterning, Lithium battery electrode forming. You can have the very best and exquisitely designed equipment for manufacture and yet without experienced experts running the process (replicating the experiment day in and day out) it will go out of control and you will get garbage in a million ways. It's like amateurs playing chess against a master... where the master is the natural variation of the environment.

High value semi manufacturing is a great example of how hard it is. The processes are designed to be as simple as possible to replicate. You replicate every day as many times as possible. You have huge amounts of money and resources to succeed and sometimes you still fail. Getting to 10% yield/replication is a big step... getting to 50% maybe harder... then 90%... and higher. Novel science is often harder and there's much less money and the time to replicate could be months/result rather than 1000s of testable results/min.

If you can't replicate that means the original research didn't document the conditions adequately.

The whole point is that some people CAN replicate, but not everyone can, because the documentation for real (interesting) experiments would be so complex for a layman that they would likely become confused and never finish. If I give you and your 20 best friends a $300M EUV machine with all the documentation in the world, the best thing you could do is sell it for $10M in parts. Give it to TSMC or Intel's experts and they will make $Bs in chips at 50% margins. YOU can't replicate the experiment. Most people can't replicate the experiment, and being unable to do so does NOT make you special. However, some people can replicate it many millions of times a year.

Who's results would you publish?

Undermining the whole concept of "science".

Perhaps with some justification, perhaps not. But if replication is impossible because of "irreducible complexity" (seems to be the argument?) then why bother with science at all?

Things are not that bad. But in some fields other approaches may be more productive.

For example clinical medicine may have more to contribute than scientific medicine. Psychological studies in particular are notorious for being impossible to replicate. Does not mean we should give up on psychology. It does not have to be science to be useful

Prediction markets are an interesting alternative solution. If people can make money by correctly predicting the outcomes of experiments, and you have a way to prevent profit-driven bias impacting outcomes (e.g. independent experiment designers and performers, and the performers can't participate in the market but maybe get a cut of the transaction fees so that more "interesting" experiments get performed), then you get a pretty different research dynamic.

(I am paraphrasing Eliezer Yuskowsky here, specifically a small part of his NSFW and frankly weird collaborative fiction Mad Investor Chaos.)

If prediction markets could help, we wouldn't need science.

Then how would you resolve the markets? The predictions would be about the outcomes of experiments...