Is novel research worth doing? Evidence from peer review

Re:Is novel research worth doing? Evidence from peer review by auggierose on Friday November 18, 2022 @08:27AM (#33654256)

In the end, trying to find a perfect system for new and original research (I like new and original much better than "novel") is not possible, because new and original research does not follow any rules. All you can do is to build enough slack into the system so that people can do whatever the hell they are interested in, in addition to their measured duties. But then how do you admit people into this system? If it is based on objective measurements, you will mostly get people who will do great at their measured duties, but will mostly not be able to use the slack to produce new and original research; instead they will turn the provided slack into objectively measurable activities. Bar unlimited resources, I don't see a good way out of this conundrum. Do you?

[ Parent | on HN ]

Re:Is novel research worth doing? Evidence from peer review by aleks224 on Friday November 18, 2022 @09:09AM (#33654912)

I believe the culprit is the assumption the grants/tenure-track systems make about applicants/assistant professors. That assumption is negative, as if new hires will try _not to do research_. It's also about the number of grad school offerings, it just seems so huge at the moment, which again forces introducing such metrics on what is considered "success" in academia.

[ Parent | on HN ]

Re:Is novel research worth doing? Evidence from peer review by auggierose on Friday November 18, 2022 @09:45AM (#33655444)

Of course researchers will want to do research. The question is, how do you select for those whose research will ever amount to something truly interesting, and avoid giving resources to those whose research will not?

You can try looking at the past, and derive criteria for it from that, via machine learning for example, but I would be hesitant to leave something like that up to a machine. Also times change, so criteria that worked in the past might not work now. Also, if you learn those criteria once, and then fix them, people will just game them.

[ Parent | on HN ]

Re:Is novel research worth doing? Evidence from peer review by Jensson on Friday November 18, 2022 @10:17AM (#33655903)

Current system is like measuring programmer productivity by lines of code so I think it is hard to do worse than what we currently do.

[ Parent | on HN ]

Re:Is novel research worth doing? Evidence from peer review by aleks224 on Friday November 18, 2022 @10:57AM (#33656593)

>The question is, how do you select for those whose research will ever amount to something truly interesting

You can't, as "truly interesting" is context dependent, changes over time and something that everyone deems as futile may become interesting - that's the point of research. You just increase the bar of entry to get people who work very hard and leave it to them to decide.

[ Parent | on HN ]

Re:Is novel research worth doing? Evidence from peer review by auggierose on Friday November 18, 2022 @11:43AM (#33657353)

Working very hard is not well-defined in this context. What does it even mean? You can work hard when you have a clear goal, let's say put those 10 barrels onto that truck over there. Or, let's write a new web browser within 2 years. Putting out 20 papers per year can also be considered working very hard.

You don't want people who work very hard. You want people who will EVENTUALLY put out new, original, and truly interesting research. HOW they do this is not up to you.

And there is a difference between truly interesting research, and just busy work research. It's not that easy to identify truly interesting research without the benefit of hindsight. It is somewhat easier to identify busy work research for an objective subject matter expert (but of course this is not 100% either, and personal preferences can definitely cloud the experts judgement).

[ Parent | on HN ]

Re:Is novel research worth doing? Evidence from peer review by guhidalg on Friday November 18, 2022 @12:40PM (#33658452)

You need people who got excellent grades in their undergrad program, who have somehow demonstrated that they like the field they’re going into academia for (as in clubs, extracurriculars, and whatnot), and who peers recommend as being likely to do novel research. The last one requires, gasp, talking to the applicant and seeing if they’re full of shit.

[ Parent | on HN ]

Re:Is novel research worth doing? Evidence from peer review by auggierose on Friday November 18, 2022 @02:35PM (#33660798)

That's already done now. People in Academia have excellent grades, and they like what they do. How would peers know about their ability to do novel research, not having done any themselves?

[ Parent | on HN ]