News for hackers. Stuff that matters.
Re:Reproducibility Crisis: The Plot Thickens by adrianN on Thursday November 12, 2015 @02:48AM (#10551834)
Given limited funds I think we should not distribute them via a lottery. That just doesn't seem like the most efficient use of the money we have.

I fully agree with point 3 though. Currently we produce PhDs waay above replacement rate. Every tenured professor produces dozens of PhDs during his career, but his tenured position only becomes available once he retires. That's just not sustainable, no matter how much money you throw at universities and research institutes.

Re:Reproducibility Crisis: The Plot Thickens by tikhonj on Thursday November 12, 2015 @04:42AM (#10552156)
Not just a lottery, but it feels like randomness should be part of the process. The mechanisms and people we have in place to make decisions are horribly imperfect, so there's a lot of noise in the system as is. Many perfectly good grants don't get funded, often for unclear reasons. Adding randomness past a certain minimum bar just makes that noise explicit and ensures that it's not biased or spuriously correlated.

This would also deemphasize the importance of narrow skills specific to writing and getting grants. Right now, understanding the grant process and granting agency is probably just as important as, you know, coming up with good proposals. Not ideal!

Among other things, this could also help academics to spread out a bit—right now, research agendas are extremely fashion-driven, in large part because of how grants are awarded. Think about it like a search algorithm which, with some probability, randomly jumps out of its current state as a way to avoid getting stuck in local maxima.

Originally, I thought of this idea as applied to college admissions. Harvard likes to talk about how it could assemble a few classes of qualified students each year, but only has room for one. At that point, it's distinctly non-obvious that the process they use to choose between similarly qualified candidates is any good; some actual randomness could have benefits. But this idea feels even more applicable to research grants.

Re:Reproducibility Crisis: The Plot Thickens by danieltillett on Thursday November 12, 2015 @04:53AM (#10552193)
Great post.

One thing about making the process (more) random is it would encourage people to take risks. When I was an academic I had to spend most of my time working on relatively low risk projects that I knew would generate results (and get me publications) or else I knew I would not get anymore grants. By removing this need I could have afforded to take risks that might have resulted in real breakthroughs. We need scientists to take risks and make the great discoveries.

Re:Reproducibility Crisis: The Plot Thickens by leoc on Thursday November 12, 2015 @05:28AM (#10552298)
> Originally, I thought of this idea as applied to college admissions. Harvard likes to talk about how it could assemble a few sets of qualified students each year, and has to choose one. At that point, it's distinctly not obvious that the process they use to choose between similarly qualified candidates is any good; some actual randomness could have some benefits.

The only problem is that there's no sign that elite universities have any intention of choosing students on merit, or through any system which is blind to factors other than merit (plus pure chance). Harvard's selection system seems to be very good at its actual intended purpose: it's more like casting Glee than making a strict assessment of merit. But for the rare university which both has a large oversupply of very good applicants and actually wishes to operate admissions on merit then yes, randomness is the answer.

Another place that it would seem to make sense to apply randomness is in performance-related pay.

Re:Reproducibility Crisis: The Plot Thickens by danieltillett on Thursday November 12, 2015 @03:00AM (#10551869)
The reason why is to break the link between track record and grant funding. It has got so extreme that it has perverted the whole of science.

It just is not possible by peer review to determine the difference between a grant that is ranked in the top 10% and one that is ranked in the top 30%. Given we only have enough money to give less than 10% of all grants money, continuing the current process is just creating massive problems.