Studies that have not been replicated should be published clearly marked as preliminary results. And then other scientists can pick those up and try to replicate them.
And institutions need to give near equal weight to replications as to original research when deciding on promotions. Should be considered every researchers responsibility to contribute to the overall field.
Replications could be matched with grants: if you receive $100,000 grant, you'd get the $100,000 you need, plus another $100,000 which you could use to publish a replication of a previous $100,000 grant. Researchers can choose which findings they replicate, but with restrictions, e.g. you can't just choose your group's previous thing.
I think if we did this, researchers would naturally be incentivized to publish experiments that are easier to replicate and of course fraud like this would be caught eventually.
I bet we could throw away half of publications tomorrow and see no effect on the actual pace of progress in science.
Ultimately, I'm not sure how to incentivize reproduction of research: it's very easy to fake a successful reproduction (you already know the results, and the original researcher will not challenge you), so you don't want to reward that too much. Whereas incentivizing failed reproductions might lead some scientists to sabotage their own reproduction efforts in ways that are subtle enough to have plausible deniability.
Proceeding by pairs is probably not enough. You probably need 5-6 replications per paper to make sure that at least one attempt is honest and competent, and make the others afraid to do the wrong thing and stand out.
I don't work in academia but in my experience professors are basically all intellectually arrogant and ego-driven, and would relish having time and space to beat each other at the brain game. A failed replication is their chance to be "the smarter guy in the room" and crack open some long-held belief. A successful replication would probably happen most of the time and be far more boring.
I could imagine, if such a thing were mandated and in place for a while, one could build her career on replications, as a prosecutor or defense. She would publish new research solely to convince her colleagues that she is sharp enough to play prosecutor or defense.
Anything has got to be better than what we have now, where apparently you can cheat and defraud your way through an entire decades-spanning career.
And those same people are reviewing the papers you publish and will not hesitate to sabotage your career if you have made them look bad by failing to replicate their papers.
Making fraud much less convenient will greatly reduce the amount of it.
If you publish a paper with fraudulent data, methods, or results, and you received any state or federal funds for it, there should be prison time. You stole taxpayer money.
I'm not saying for when people are wrong, I'm saying for when you can prove someone knowingly lied. It won't catch anyone, and you need to bar to be high enough that people don't go to jail for being bad scientists, but right now there is zero social, professional, or legal risk is just lying your ass off to get the next grant and keep the spice flowing.
Nobody's going to do that when changing the numbers in your Excel sheet carries a risk of a decade or two in a minimum security prison.
It might actually improve the pace of science, if the half eliminated were not replicable and the remaining half were written by researchers knowing that they would likely face a replication attempt.
Why go through all the work of actually proving/disproving the experiment when you can just change tweak the numbers of the original experiment, say you actually reproduced the experiment, and then move on?
And get a nice $100k too for close to zero effort!
(Having in mind: "plus another $100,000 which you could use to publish a replication")
But if there is separate funding for replication studies, it will become something people compete for. Some people will specialize on replicating others' work, and universities will pay attention, as they care about grant overheads.
It would need to be very good funding on par with what's offered for "novel research".
In addition, we would need increased prestige (e.g. awards, citations) for replicated studies as well for this to be effective. For many academics funding is merely a means to that end.
You can avoid that to some extent by having replication and original work be separate specialities - and making sure that replication gets prestige so good people do it.
But the biggest problem is actually that large swathes of 'scientists' don't do experiments at all. You can't even replicate such papers because they exist purely in the realm of the theoretical. The theory often isn't even properly written down! They will tell you that the paper is just a summary of the real model, which is (at best) found in a giant pile of C or R on some github repo that contains a single commit. Try to replicate their model from the paper, there isn't enough detail to do so. Try to replicate from the code, all you're doing is pointlessly rewriting code that already exists (proves nothing). Try to re-derive their methodology from the original question and if you can't, they'll just reject your paper as illegitimate criticism and say it wasn't a real replication.
Having reviewed quite a lot of scientific papers in the past six years or so, the ones that were really problematic couldn't have been fixed with incentivized replication.
If a finding either cannot be communicated enough for someone else to replicate it, or cannot be replicated because the method is shoddy, can we even call that science?
At some level I know that what I'm proposing isn't realistic because the majority of science is sloppy. P-hacking, lack of detail, bad writing, bad methods, code that doesn't compile, fraud. But maybe if we tried some version of this, it would cause a course correction. Reviewers, knowing that someone actually would attempt to replicate a paper at some point down the road, would be far more critical of ambiguity and lack of detail.
Papers that are not fit to be replicated in the future, whose claims cannot be tested independently, are actually not science at all. They are worth less than nothing because they take up air in the room, choking out actual progress.
The amazing thing is that it all works out in the end and science is still making (quite a lot of) progress.
That's also the reason why we shouldn't spend all of our time and money checking and replicating things just to make sure noone publishes fraudulent/shoddy results. (We should probably spend a little more time and money on that, but not as much more as some people here seem to suggest).
Most research is in retrospect useless nonsense. It's just impossible to tell in advance. There is no point in checking and replicating all of it. Results that are useful or important will be checked and replicated eventually. If they turn out to be wrong (which is still quite rare), a lot of effort is wasted. However, again, that's rare.
If the fraud/quality issues get worse (different from "featuring more frequently and prominently in the news"), eventually additional checks start to make sense and be worth it overall. I think quite a lot of progress is happening here already, with open data, code, pre-registration of studies, better statistical methods, etc, becoming more common.
I think a major issue is the idea that "papers are the incontestable scientific truth". Some people seem to think that's the goal, or that it used to be the case and fraud is changing that now, however, this was never the case and it's not at all the point of publishing research. I think a major gain would be to separate in the public perception the concepts, understanding and reputations of science vs. scientific publishing.
Why are these obvious low value papers a) getting grants, b) getting published, c) not permanently damaging the researchers' careers?
If you do bad work you eventually get fired, why don't we do the same thing with research academics who do bad work?
Undoubtedly this is true. The problem, like with advertising, is identifying _which_ half to cut.
Many of the algorithms I tried to implement didn't work at all, despite considerable effort to get them to behave. In one particularly egregious (and highly cited) example, the algorithm in the paper differed from the provided code on GitHub. I emailed the authors trying to figure out what was going wrong, and they tried to get funding from me for support.
My manager wanted me to right a literature review paper which skewered all of these bad papers, but I refused since I thought it would hurt my career. Ironically the algorithm that ended up working the best was from one of the more unknown papers, with few citations.
I do like the idea of institutions giving tenure to people with results that have stood the test of time, but again, there is no incentive to do so. Institutions want superstar faculty, they care less about whether the results are true.
The only real incentive that I think can be targeted is still grant money, but I would love to be proved wrong.
This one is the showstopper. No matter what you do with rules and regulations, if people aren't impressed by it at a watercooler conversation, or when chatting at a cocktail party at a conference, or when showing a politician around in your lab then nothing else matters.
How prestigious something is is not a lever you control.
Prestige is a nice motivator, but making a living at all is always the baseline, and is often sufficient.
What we should aim for is confirmation: a different experiment that tests the underlying phenomenon that was the subject of the first paper.
Building more results without replications is what caused the psychology crisis. Apparently every lab accepted the p<0.05 results or stated correlations of prior studies and just ran more studies until they got their own that was publishable. Since everyone "knew" that the prior result was true, like priming or whatever, they could conclude anything they wanted, because ex absurdum quodlibet.
If the published work under specifies the experiment such that it is unreproducible, that means the results can’t be reliably extrapolated because there are unstated conditions.
No researchers are going to invest the time needed to replicate complex neurobiological experiments such as those Masliah conducted.
However, if the results are sound, others will be able to build upon them, and this happens a lot more often than fraud does.
This is a very difficult problem to solve.
Basically, I think there is a sizable risk that a large number of replications would be fraudulent or half-assed, which dilutes their value. Paradoxically, the more this policy suppresses fraud or mistakes in original research, the less people will perform replication in good faith.
I could be wrong, but people are endlessly creative at subverting systems when the stakes are high, so I'm wary of simple solutions. To be fair, it's probably better than the current system, just not as much as we'd like.
unless there are grants specifically for that purpose, then it's not going to happen; and it's hard to apply for a grant just to replicate someone else's results verbatim. (usually you're testing the theory but with a different experiment and set of data which is much more interesting than simply repeating what they did with their data; in fact replicating it with a different set of data is important in order to see if the results weren't cherry-picked to fit the original dataset).
If the older author is your advisor and you feel one of their juniors is cutting corners or the elder is cutting corners, you better think twice about what move will help your career. If confirming a recent result counts toward tenure, then presto you have an incentive for fraudulent replication (what's the chance it's incorrect anyway? The original author is a big shot.) Going against the previous acclaimed result takes guts especially in a small field where it might kill your career if YOU got it wrong somehow - So you need to have much stronger results than the original research, and good luck with that. We might say "this is perfect work for aspiring student researchers, and done all the time" - to reimplement some legendary science experiment - but no, not when it's a leading edge poorly understood experiment, and not when that same grad student is already running to try and produce original research themselves.
The big funders might dedicate money to replicated research that everybody is enthusiastic about (before everyone relies on it). But some research takes years to run. Other research is at the edge of what's possible. Other research is led by a big shot nobody dares to take on. Etc etc. So where is the incentive then? The incentive might be to take the money, fully intending to return an inconclusive result.
Some research is taken on now. But only AFTER it's relied on by lots of people. Or much later when better ideas had the time to emerge on how to test the idea more cleverly i.e. cheaper and faster. And that's not great because costly in all the wasted effort by others, based on a fraudulent result. And all the mindshare the bad result now has.
This is messed up.
Yes and ... we're already there.
Removing the government from science could help, not the other way around.
People just went through the last five years and will go to their graves defending what they saw first hand. To admit that maybe those moves and omissions weren’t helpful would be to admit their ideology was wrong. And that can not be.
But there was no golden era of science to look back on. It has always been a wonderful productive mess—much like the rest of life. As least it moves forward—and now exceedingly rapidly.
Almost unbelievably, there are far worse crimes than fraud that we completely ignore.
There are crimes associated with social convention in science of the type discussed by Karl Herrup with respect to 20 years of misguided focus on APP and abeta fragments in Alzheimer’s disease:
https://mitpress.mit.edu/9780262546010/how-not-to-study-a-di...
This could be called the “misdemeanors of scientific social inertia”. Or the “old boys network”.
There is also an invisible but insidious crime of data evaporation. Almost no funders will fund data preservation. Even genomics struggles but is way ahead in biomedical research. Neuroscience is pathetic in this regard (and I chaired the Society for Neuroscience’s Neuroinformatics Committee).
I have a talk on this socio-political crime of data evaporation.
https://www.youtube.com/watch?v=4ZhnXU8gV44&embeds_referring...
If someone brings a plum to a market for lemons, they can distinguish the quality of their product by offering a warranty on its purchase, something that sellers of lemons would be unwilling to do, because they want to pass the cost burden of the lemon onto the purchaser.
The full paper is fairly accessible, and worth a read.
Not sure how this could be applied to academia, one of the problems is that there can be significant gaps between perpetrating fraud and having it discovered, so the violators might still have an incentive to cheat.
Is this really the case the though? Isn't the whole point of tenure (or a big selling point at least) insulating academics from capricious firings?
The big question I have is that there are names on these fraudulent papers, so why are these people still employed? If you generate fictitious data to get published, you should lose any research or teaching job you have, and have to work at McDonald's or a warehouse for the rest of your life. There are plenty of people who want to be professors that we can eliminate the ones who will lie while doing it without losing much (perhaps anything). If your job was funded by taxpayer funds there should be criminal charges associated with willfully and knowingly fabricating data, results, or methods. At that point you're literally lying in order to steal taxpayer funds, it's no different than a city manager embezzling or grabbing a stack of $20 bills out of the cash register.
That, and you select for the kind of people who are willing to fake results to further their own careers.
Most types of fraud carry a $100-250k monetary penalty and up to 20, 25, 30 years in prison.
The number of people willing to fabricate research data decreases dramatically if you're going to have to pay the grant back from your $20/hr warehouse job after you spend the better part of a decade in a minimum security prison.
Bear in mind also that the vast majority of academic fraud isn’t cut and dried easily proved. It’s p hacking or “accidental” flaws in an analysis, or forgetting to mention some important detail.
The situation is similar to the "Market for lemons" in cars: if the market is polluted with lemons (fake papers), you are disincentivized to publish a plum (real results), since no one can tell it's not faked. You are instead incentivized to take a plum straight to industry and not disseminate it at all. Pharma companies are already known to closely guard their most promising data/results.
Similar to the lemon market in cars, I think the only solution is government regulation. In fact, it would be a lot easier than passing lemon laws since most labs already get their funding from the government! Prior retractions should have significant negative impact on grant scores. This would not only incentivize labs, but would also incentivize institutions to hire clean scientists since they have higher grant earning potential.