Most professors I have worked with in experimental sciences are completely disconnected from actual research. They just optimize for hiring hordes of people to churn out tons of mediocre papers, which they don't care about. I came from theoretical CS & math, so this was really shocking.
It's really depressing and it needs to change. Academia is now in a phase similar to that of an innovative startup which has been filled in with many middle managers, and has turned into a corporate monstrosity.
And while publications do help you, say in getting hired and promotions, your publications are dug into pretty deeply. They don't want to be tricked into hiring someone useless who can crank out fluff but can't win a grant.
By whatever metric you use, it's really about the checklist rather than substance. And the checklists are based on stereotypes, and get gamed.
Indeed in some ways it's worse. When the money is insufficient, the established researchers get it first. Then everyone else has to increasingly work for established researchers as postdocs and similar positions... helping to yet further boost the advantage of the established researchers. It went from senior researchers versus junior researchers, to senior researchers with a herd of postdocs working for them, versus junior researchers.
At least if it was based on publications, everyone gets co-authorship and in fact the first author counts most. So the postdoc and the PI gain the same new items on their CV's for their collaboration. For grants, credit goes entirely to the PI.
In software, we have "source code distributions", such as Debian, Fedora, Gentoo, Arch, Red Hat, SUSE, etc. They are collections of mostly a same free software, with minor differences, but with different policies and purpose. Some are conservative collections of proven software, others are bleeding edge, etc.
IMHO, someone should start to collect papers, which are worth reading, and keep this collection up to date, with fixes to papers (AKA as patches in software), with one paper per topic, with ready to use math models and simulations, with in software tests for models, etc.
I am interested in Amyotrophic Lateral Disease and I blog about research done on ALS [0].
However there are 15,000 scientific papers published each year on this topic, on average only 20 to 30 articles per year are worth spending the time to read them (one every 500) and a milestone appears only every 3/4 years (TDP-43 in 2006, C9orf72 in 2011, etc).
After 100 years there is still no clue about ALS etiology. Scientists have strong but widely different opinions such as for some it starts in the brain motor area, for other it starts at the peripheral nerve junction with muscles, for other (new trend) it starts in the gut.
And scientists happily pursue hypothesis that their colleagues are supposed to have disproved. Sometimes the same lab (I have one in mind which is perhaps the best known in ALS research) publishes articles contradicting its previous research without discussing the issue.
How to rationally blog about this domain, how to make collections that are worth reading?
Every year there are 15 gazillion modules published to NPM, on average only 20-30 are worth spending the time to read, and only a handful milestones worth including in said distributions.
After 100 years there is still no clue weather microservices or monoliths are the correct hypothesis and some times the same lab publish articles contradicting previous research.
IMHO, something similar can be applied to science. Leaders can chose a general direction of a collection of papers, while maintainers will work on small areas, keeping them consistent and up to date with new research.
For example:
- LIGO/VIRGO discovery of gravitational waves means that the speed of light in vacuum is not constant. It's constant in "steady" vacuum only.
- Discovery of Higgs boson means that vacuum is not an empty space, because Higgs field is present everywhere.
These recent discoveries significantly changes the field of science, so a bunch of classic paper must be updated or discarded in favor of new ones.
This still describes a lot of software engineering.
It has been like that since ten years and nothing has changed since then. People find out just about the same as you did but nothing changes. It such a convenient metric it will stay for much longer.
I think of it as a kind of predatory or scavenger behavior. They basically steal the citations from the papers they are reviewing in return for a small effort in helping us with information retrieval. We might as well just cite google and elsevier. Ideally, citations should pass through them somehow and get applied to the original sources who did the hard work.
Even the summary papers can be useful. On particularly active problems there can be dozens of papers per year. Sorting out wheat from chaff by highlighting notable works, and pointing out trends in published research, is again a genuine contribution, and helpful for the scientific community.
These are often very valuable, more so than yet another 1% improvement paper that never gets reproduced.
Review papers don't steal citations. These are cited from more distant literature so readers can familiarize themselves with the topic. Those wouldn't cite all the papers mentioned in the review individually.
But as a broader point, yeah, maybe something like PageRank could be used to "pass on" the citation Ina sense.
It's not an either-or.
And one should always read and accurately describe the contents of what you cite, meaning of course you wouldn't cite everything in a review paper directly. Only that which is relevant.
Suppose we count both separately, research citations and review citations, the latter for your great review papers or even if people just want to reference the lit review section of your research paper. What would happen to the incentive? I suggest that reviews would be less incentivized as people would primarily be concerned with rating researchers according to their research citations. Just like no one cares about your textbook sales, popular though it may be for introductory uses. This would imply reviews were not "honest" citations but are done to hack the metric.
Reviews also cause other devious problems, like journals hacking their own impact factors. And don't forget the Matthew principle (aka rich get richer) whereby only big-shot researchers can get invited to have their reviews published in high-impact journals, warping the network effects to their favor even more.
A related problem I see is that we treat all papers the same: position (opinion) papers, reviews & surveys, and "hard" results (relevant proofs, strong empirical results, software artifacts that are deployed at scale in practice etc.). As the blog post somewhat suggests, the administration should primarily ask us to summarize these results when assessing whether we are worthy of funding and faculty positions; sure, it takes more effort, but I think it is possible.