Having said this we could grandfather in the current “winners” and move the funding to the new system over a long time period (say 20 years). We really need to do something as the current system will destroy science if we let it continue.
I'm uncertain about this. The scientific method can luckily not be destroyed, and I wonder whether the self-destruction of academia is a good or bad thing. It is certainly not painless and will certainly take innocent victims.
If you consider non-equilibrium thermodynamics analogies, the quicker the system destroys itself the better it is, as it more or less _has to_ explode before it reorganises.
In Finland it has started already, as universities are required to get part of their funding from industry, and it's especially hard to accomplish in smaller cities.
I fully agree with point 3 though. Currently we produce PhDs waay above replacement rate. Every tenured professor produces dozens of PhDs during his career, but his tenured position only becomes available once he retires. That's just not sustainable, no matter how much money you throw at universities and research institutes.
This would also deemphasize the importance of narrow skills specific to writing and getting grants. Right now, understanding the grant process and granting agency is probably just as important as, you know, coming up with good proposals. Not ideal!
Among other things, this could also help academics to spread out a bit—right now, research agendas are extremely fashion-driven, in large part because of how grants are awarded. Think about it like a search algorithm which, with some probability, randomly jumps out of its current state as a way to avoid getting stuck in local maxima.
Originally, I thought of this idea as applied to college admissions. Harvard likes to talk about how it could assemble a few classes of qualified students each year, but only has room for one. At that point, it's distinctly non-obvious that the process they use to choose between similarly qualified candidates is any good; some actual randomness could have benefits. But this idea feels even more applicable to research grants.
One thing about making the process (more) random is it would encourage people to take risks. When I was an academic I had to spend most of my time working on relatively low risk projects that I knew would generate results (and get me publications) or else I knew I would not get anymore grants. By removing this need I could have afforded to take risks that might have resulted in real breakthroughs. We need scientists to take risks and make the great discoveries.
The only problem is that there's no sign that elite universities have any intention of choosing students on merit, or through any system which is blind to factors other than merit (plus pure chance). Harvard's selection system seems to be very good at its actual intended purpose: it's more like casting Glee than making a strict assessment of merit. But for the rare university which both has a large oversupply of very good applicants and actually wishes to operate admissions on merit then yes, randomness is the answer.
Another place that it would seem to make sense to apply randomness is in performance-related pay.
It just is not possible by peer review to determine the difference between a grant that is ranked in the top 10% and one that is ranked in the top 30%. Given we only have enough money to give less than 10% of all grants money, continuing the current process is just creating massive problems.
Of course, that would mean even less grants...
5. Each PI should have a single grant at a time (and a long one), to end the "money competition" (that mirrors the publication competition) and spread the grants better.
If we were to move to a lottery process then there would be no need to limit anyone.
> This particular plot is a statistical smoking gun, and suggests that the positive results from the original studies (black dots) were probably the result of p-hacking. They were chance findings, selectively published because they were positive.
From that, it seems to me like the problem isn't what the researchers published, it's what they didn't publish.
I'm a programmer, not a researcher, but in college chemistry we had marble notebooks that we wrote everything down in. The notebooks made it obvious if a page was torn out, everything had to be written in pen, and if we crossed something out, we had to cross it out in such a way that it was still legible. If we didn't include data in our results, we were expected to explain why. The most valuable thing I learned in those classes was the importance and practice of auditability.
I doubt that my classes were unique in this respect and I wonder if this might be the solution to the problem. I've long been a proponent of people simply publishing literally all their data now that internet hosting is so cheap, and only publishing syntheses of the data in journals, giving anyone the opportunity to verify the author's conclusions. Psychology is different due to the need for confidentiality for the people being studied, but the full data should be available to at least the peer reviewers before a study can be published.
EDIT: To add to this, elsewhere on this thread there's a lot of discussion about people not publishing negative results because they're not interesting. But reality doesn't have a bias towards results that are interesting, only humans do. I don't know how to solve this problem, but at least making publication of your non-interesting results a condition of publishing your interesting results would be a start.
I think the problem is that the metrics which academics are competing to maximize are poor. If your metric is "How much interesting science has this person published?" you're going to get very different results from "How much accurate science has this person published?" Reducing competition doesn't change which question is being asked, it just reduces the importance of any metric at all. It seems likely to me that removing competition would give no incentive to publish anything measurable at all; after all, why publish if your career isn't connected to it?
Obviously some would publish for intrinsic reasons, and it's possible that stress levels due to competition are leading to shoddy work, but none of this amounts to an argument that less competition would lead to people publishing negative results.
Why can't people simply publish negative results?
> Break the link between publication and grant funding.
Same answer - in fact give extra priority to those who publish lots of negative results.
Mainly because it's not as interesting. For every positive result researchers find there are dozens of things they test that just don't work.
Imagine if the first test of "romantic priming" didn't work. Would the researchers really bother trying to publish "Hypothesised romantic priming effect was just a wild hunch after all."? And would the journal accept it? No way.
Of course if you get a negative effect for an established "fact", as in this case it can be published. But you have to be really really sure otherwise the reviewers will just say you did your experiment wrong.
That seems to be the core of the problem. Reality doesn't have a bias toward results that are interesting, humans do.
> Imagine if the first test of "romantic priming" didn't work. Would the researchers really bother trying to publish "Hypothesised romantic priming effect was just a wild hunch after all."? And would the journal accept it? No way.
So maybe they should.
The journal's peer review process tends to reject them. Or so I'm told by a researcher friend of mine.
In addition to the other comments: because a negative result is usually much more likely than a negative one, it is very hard to be sure that the experimenter did not simply screw up the experiment.
A lack of positive proof generally does not constitute a negative proof.
There is only so much money for scientific research as all areas of science are vast. Is it really surprising that there is so much pressure?
5. Don't limit review to peer review, but involve people from different fields. What can be a well known source of bias in one field may be overlooked in another.
If we are going to solve this problem we need to reduce the pressure on scientists to publish at all costs. This publish-or-die thinking is driven by the extreme competition for grants and tenure tracked positions. Given I think it unlikely that the amount of money available for research is going to quadruple anytime soon (this is about the amount of money we would need to put into the system to get rid of the publish-at-all-costs mentality) what can be done?
1. Break the link between publication and grant funding. We could move to a system where once you had proved that you could do research (say published a few papers) and could come up with a viable research idea that was doable assessed by a simple grant proposal, then you go into a pool and the grants allocated by lottery. We should aim to have 50% to 75% of all scientists in this pool. We then fund as many grants as we can given the money available.
2. Spend some of the current research money on long-term fellowships (say 10 years) that are awarded in the same lottery way as grants.
3. Drastically reduce PhD student training to get supply and demand back into line. e.g. make it such that a supervisor can not supervise more than one student at a time.