A related problem I see is that we treat all papers the same: position (opinion) papers, reviews & surveys, and "hard" results (relevant proofs, strong empirical results, software artifacts that are deployed at scale in practice etc.). As the blog post somewhat suggests, the administration should primarily ask us to summarize these results when assessing whether we are worthy of funding and faculty positions; sure, it takes more effort, but I think it is possible.
Most professors I have worked with in experimental sciences are completely disconnected from actual research. They just optimize for hiring hordes of people to churn out tons of mediocre papers, which they don't care about. I came from theoretical CS & math, so this was really shocking.
It's really depressing and it needs to change. Academia is now in a phase similar to that of an innovative startup which has been filled in with many middle managers, and has turned into a corporate monstrosity.
And while publications do help you, say in getting hired and promotions, your publications are dug into pretty deeply. They don't want to be tricked into hiring someone useless who can crank out fluff but can't win a grant.
By whatever metric you use, it's really about the checklist rather than substance. And the checklists are based on stereotypes, and get gamed.
Indeed in some ways it's worse. When the money is insufficient, the established researchers get it first. Then everyone else has to increasingly work for established researchers as postdocs and similar positions... helping to yet further boost the advantage of the established researchers. It went from senior researchers versus junior researchers, to senior researchers with a herd of postdocs working for them, versus junior researchers.
At least if it was based on publications, everyone gets co-authorship and in fact the first author counts most. So the postdoc and the PI gain the same new items on their CV's for their collaboration. For grants, credit goes entirely to the PI.
In software, we have "source code distributions", such as Debian, Fedora, Gentoo, Arch, Red Hat, SUSE, etc. They are collections of mostly a same free software, with minor differences, but with different policies and purpose. Some are conservative collections of proven software, others are bleeding edge, etc.
IMHO, someone should start to collect papers, which are worth reading, and keep this collection up to date, with fixes to papers (AKA as patches in software), with one paper per topic, with ready to use math models and simulations, with in software tests for models, etc.
I am interested in Amyotrophic Lateral Disease and I blog about research done on ALS [0].
However there are 15,000 scientific papers published each year on this topic, on average only 20 to 30 articles per year are worth spending the time to read them (one every 500) and a milestone appears only every 3/4 years (TDP-43 in 2006, C9orf72 in 2011, etc).
After 100 years there is still no clue about ALS etiology. Scientists have strong but widely different opinions such as for some it starts in the brain motor area, for other it starts at the peripheral nerve junction with muscles, for other (new trend) it starts in the gut.
And scientists happily pursue hypothesis that their colleagues are supposed to have disproved. Sometimes the same lab (I have one in mind which is perhaps the best known in ALS research) publishes articles contradicting its previous research without discussing the issue.
How to rationally blog about this domain, how to make collections that are worth reading?
Every year there are 15 gazillion modules published to NPM, on average only 20-30 are worth spending the time to read, and only a handful milestones worth including in said distributions.
After 100 years there is still no clue weather microservices or monoliths are the correct hypothesis and some times the same lab publish articles contradicting previous research.
IMHO, something similar can be applied to science. Leaders can chose a general direction of a collection of papers, while maintainers will work on small areas, keeping them consistent and up to date with new research.
For example:
- LIGO/VIRGO discovery of gravitational waves means that the speed of light in vacuum is not constant. It's constant in "steady" vacuum only.
- Discovery of Higgs boson means that vacuum is not an empty space, because Higgs field is present everywhere.
These recent discoveries significantly changes the field of science, so a bunch of classic paper must be updated or discarded in favor of new ones.
This still describes a lot of software engineering.
It has been like that since ten years and nothing has changed since then. People find out just about the same as you did but nothing changes. It such a convenient metric it will stay for much longer.
I think of it as a kind of predatory or scavenger behavior. They basically steal the citations from the papers they are reviewing in return for a small effort in helping us with information retrieval. We might as well just cite google and elsevier. Ideally, citations should pass through them somehow and get applied to the original sources who did the hard work.
Even the summary papers can be useful. On particularly active problems there can be dozens of papers per year. Sorting out wheat from chaff by highlighting notable works, and pointing out trends in published research, is again a genuine contribution, and helpful for the scientific community.
These are often very valuable, more so than yet another 1% improvement paper that never gets reproduced.
Review papers don't steal citations. These are cited from more distant literature so readers can familiarize themselves with the topic. Those wouldn't cite all the papers mentioned in the review individually.
But as a broader point, yeah, maybe something like PageRank could be used to "pass on" the citation Ina sense.
It's not an either-or.
And one should always read and accurately describe the contents of what you cite, meaning of course you wouldn't cite everything in a review paper directly. Only that which is relevant.
Suppose we count both separately, research citations and review citations, the latter for your great review papers or even if people just want to reference the lit review section of your research paper. What would happen to the incentive? I suggest that reviews would be less incentivized as people would primarily be concerned with rating researchers according to their research citations. Just like no one cares about your textbook sales, popular though it may be for introductory uses. This would imply reviews were not "honest" citations but are done to hack the metric.
Reviews also cause other devious problems, like journals hacking their own impact factors. And don't forget the Matthew principle (aka rich get richer) whereby only big-shot researchers can get invited to have their reviews published in high-impact journals, warping the network effects to their favor even more.
The flip-side to this is that there is a glut of moderately motivated scientists and PhD candidates. As a result, the demand for conferences and citations is self-sustaining to meet the expectations of millions of people wanting to enter the academic world without a clear plan beyond joining the world itself.
It's as though the English-major-to-professor cycle has also become a reality in the hard sciences.
“If you pay a man a salary for doing research, he and you will want to have something to point to at the end of the year to show that the money has not been wasted. In promising work of the highest class, however, results do not come in this regular fashion, in fact years may pass without any tangible result being obtained, and the position of the paid worker would be very embarrassing and he would naturally take to work on a lower, or at any rate a different plane where he could be sure of getting year by year tangible results which would justify his salary. The position is this: You want one kind of research, but, if you pay a man to do it, it will drive him to research of a different kind. The only thing to do is to pay him for doing something else and give him enough leisure to do research for the love of it."
[1] https://archive.org/details/b29932208 - see pages 198-200, ref HN comment: https://news.ycombinator.com/item?id=21388715
On a speculative note: I believe it's yet another argument in favor of UBI
As parent said, UBI would probably fix the incentive scheme, your solution wouldn't. You need to decouple the reward from the result so that research is done out of sheer curiosity and love of science.
It's such a mess at the moment that it's not really possible to continue out of inertia. Even if you're well-suited to research and discovery at some level, and motivated, it's a soul-crushing experience because at every step you're incentivized toward something else.
In that essay, Prof. Goodstein explains how academic had been growing exponentially for decades since WWII, with each PhD giving birth to another litter of fifteen PhDs (my phrasing...), until finally that exponential process began to hit limits. That set of changes from the ending of the exponential growth in academia beginning around the 1970, leading to a break down of peer review and lots of other issues.
That's also part of why those of us (like me) who got honest well-meant advice from extremely successful academics in the 1980s or later about the value of a PhD and a career in academia were misled, even though that advice had worked for the advice giver. Freeman Dyson, by contrast, tried to steer bright people (including me) away from PhDs even in the 1980s -- having an intuition for this sort of thing. Wish I had listened to him more. :-)
Also related by Philip Greenspun from 2006 (later updated), on "Women in Science" but which also applies more broadly and is a good description of the consequences of the situation Goodstein described in 1994: http://philip.greenspun.com/careers/women-in-science "This is how things are likely to go for the smartest kid you sat next to in college. He got into Stanford for graduate school. He got a postdoc at MIT. His experiment worked out and he was therefore fortunate to land a job at University of California, Irvine. But at the end of the day, his research wasn't quite interesting or topical enough that the university wanted to commit to paying him a salary for the rest of his life. He is now 44 years old, with a family to feed, and looking for job with a "second rate has-been" label on his forehead. Why then, does anyone think that science is a sufficiently good career that people should debate who is privileged enough to work at it? Sample bias. ... A good career is one that pays well, in which you have a broad choice of full-time and part-time jobs, in which there is some sort of barrier to entry so that you won't have to compete with a lot of other applicants, in which there are good jobs in every part of the country and internationally, and in which you can enjoy job security in middle age and not be driven out by young people willing to work 100 hours per week. How closely does academic science match these criteria? I took a 17-year-old Argentine girl on a tour of the M.I.T. campus. She had no idea what she wanted to do with her life, so maybe this was a good time to show her the possibilities in female nerddom. While walking around, we ran into a woman who recently completed a Ph.D. in Aero/Astro, probably the most rigorous engineering department at MIT. What did the woman engineer say to the 17-year-old? "I'm not sure if I'll be able to get any job at all. There are only about 10 universities that hire people in my area and the last one to have a job opening had more than 800 applicants." And that's engineering, which, thanks to its reputation for dullness and the demand from industrial employers, has a lot less competition for jobs than in science. What about personal experience? The women that I know who have the IQ, education, and drive to make it as professors at top schools are, by and large, working as professionals and making 2.5-5X what a university professor makes and they do not subject themselves to the risk of being fired. With their extra income, they invest in child care resources and help around the house so that they are able to have kids while continuing to ascend in their careers. The women I know who are university professors, by and large, are unmarried and childless. By the time they get tenure, they are on the verge of infertility. ... I've taught a fair number of women students in electrical engineering and computer science classes over the years. I can give you a list of the ones who had the best heads on their shoulders and were the most thoughtful about planning out the rest of their lives. Their names are on files in my "medical school recommendations" directory. ..."
Anyway, that's all part of why I support a UBI. Give everyone enough money to live as a perpetual graduate student (without publishing and without becoming a "Disciplined Mind" as the name of a book by Jeff Schmidt on academic perils) and some of them will, which will eventually remake academia in a healthier way (including taking more chances on small-scale but far reaching basic research, whether towards cold fusion, quantum teleportation, new computer languages, new materials, new batteries, new ways of thinking, new communications patterns, new ways of conflict resolution, or whatever). If for that reason alone the USA should invest in a Universal Basic Income.
Maybe there is a clue in the name "re-search". Epistemologically I don't think it is possible to set out to discover something that isn't already known to exist, so the justification for paying people would best be for evidence that they were searching rather than if they find something.
This year, less than 10% of NHMRC grant applications got approved, and of the ones approved most spent nearly as much time on grant applications as they did on the real work. Anyone who is only moderately motivated has retrained, because working for low pay half of the year is not appealing to anyone who can pass any doctorate program.
Even if you took the position that only the good ones are getting funded, spending 50% of your 'good' researchers time is a tragic waste of human ability.
One answer to all these propositions is that the valuable stuff is still there, you just have an additional flood of mediocre boring predictable stuff.
I don't think this is true for most things with a mass market. I think products which have efficiencies of scale - or low marginal costs of reproduction - suck up more capital, which raises certain basic levels of standards and polish and marketing, but also greatly increases capital risk and thus reduces investor risk appetite for product novelty. These mass market products starve the mid-market of attention, which makes the mid-market less capital efficient, which means the mid-market needs to get cheaper and cheaper, lose polish, until it's less attractive to the mass market of consumers, and so on in a spiral until a certain spartan rawness becomes its own aesthetic - indie games, indie films, indie music.
I think there's a concrete example which demonstrates the actual absence of that long tail of interesting content, and not just that it's harder to find: blogs.
Blogs were great in the mid 2000s. The market did two things: it matured - the best blogs got more mindshare and turned into something closer to media organizations, much more capital intensive with full time paid writers, while and Facebook and Twitter sucked up the mass market of newsfeed consumers, both on the producer and consumer side, making it easier to produce low-value inanities and easier to consume tidbits from celebrities. This sucked the air of attention from blogs; being long-form, they were too hard to write and too long to read; and being unpaid, they didn't have slick editors or snazzy design. So a lot of them shrivelled up. There are still niche blogs, plenty of them, but a lot fewer - and we can track this loss, because we have (or had) them in our newsreaders.
But there's no direct trading going on at that state to ensure they deliver what is claimed, the research "product" is just dumped into papers. And those papers serve as an impediment to subsequent researchers, since you can't publish or get funded for the same thing that's already been done and supposedly proven by someone else. Of course you wouldn't have to if they had really solved the problem. But they just threw together some minimal effort to stick their name on it first before moving on to the next low-hanging fruit somewhere else. Doing a literature review can be really frustrating.
There is a small and somewhat obscure but relatively stable middle - low-budget, high-quality videos. I attribute its survival to two things: some live off Patreon donations and occasional sponsorship deals, others live because the creators have an actual dayjob, and they do YouTube as a hobby. This is similar to blogs - best ones are done on the side, with no expectation of revenue.
The cost of evaluating each remains fairly constant, and at some point production outstrips the ability of even extremely-interested/dedicated individuals to have a good handle on it all. Beyond this point, you either need organized collective effort (and trust) to efficiently distribute the work of keeping up (and synthesize the results) or the larger community will be wasting ever-increasing slices of potential energy to extract increasingly less knowledge of the scope of activity.
But there's a difference between simple volume growth for good reasons (i.e., people chasing incentives that are aligned at multiple levels of society), and volume growth caused by people chasing incentives that are misaligned at one or more points up the stack.
If the researchers are spending their time on things even they find boring because Goodhart's law, I think it is important to recognize it and try to undo the misaligned incentives. They're not only externalizing a cost on their entire field of knowledge--but we're also collectively suffering some ill-defined opportunity cost of whatever it was they would've spent their time on if they were following curiosity.
In other words, academia is sowing the seeds of its own demise, and with it, the official institutional pinnacle of our culture [please don't wince too hard when reading that!].
----
The solution i think is less research but more library work. See https://blog.khinsen.net/posts/2020/07/08/the-landscapes-of-... and https://hapgood.us/2015/10/17/the-garden-and-the-stream-a-te... which it cites.
The amount of work library work we are not doing is just staggering. Consider these claims:
- There should be an accredited-author-ornly "kernel wikipedia" which the main one can choose to incorporate, almost like release vs staging branches.
- Different subfield way wish to keep their own intentionally biased corpuses, like https://ncatlab.org/nlab/
- After 6 months, virtually no one should have to read the original research article, because the librarian core will have incorporated it's results claims (however controversial) into the appropriate articles
- Textbooks / "knowledge bootstrap plans" should be continuously updated from the encyclopedia, not unlike how bootstrapping is managed in a package repo
- Librarians do the work, but also adjudicate disputes, as researchers will be naturally incentivized to contest how their work is incorporated as that is the primary way it is consumed.
I agree there is not enough library work and there is too much focus on the 4-15 page publication format. This amount is often not enough to fully flesh out an idea or to present something non-incremental. Rather, people tend to chunk up work, called salami publishing.
Now, sure doctoral theses also exist but the vast majority of research is presented in about 4-15 pages.
But I disagree that after 6 months everything should be integrated into other summary work.
Most published research is not worthy of being integrated into other work and 6 months are jót enough to decide. Most papers are forgotten and rightfully so. Publication doesn't mean it's correct or worthy of eternal remembrance. It just means that a scientist wants to share a finding with the research community. It's not established knowledge yet. Only when it is actually adopted and used successfully by the community, will it be part of established shared knowledge. After some years such key ideas do get written into textbooks.
This is roughly what the Citizendium project was aiming for. Sadly it seems it's not doing well these days.
This is first order thinking. Yes, if an increasingly large quantity of academic research is becoming bullshit, eventually the ratio of good to bad research will reach an inflection point where policy will be set that reduces research funding.
This reduction in funding will have many effects, but the largest will be that as funding becomes a scarce resource, a higher curation of both researchers and topics will come into effect. There will be less research and papers overall, but the largest decrease in research will be those that are bullshit.
Finally, as more and more research will be fruitful and useful, and fewer and fewer bullshit papers are published, an inflection point will be reached. There will be more funding allocated to these exciting new areas of study, and more research will happen. Of course, with funding no longer a scarce resource, less curation will occur, and a few more bullshit papers will be published... And the cycle will have advanced one wavelength.
It's been happening for years, centuries, as a basic economic cycle, and looks something like this: _/T\_/T\_/T\
> Question. In what ways can a recession be useful for forcing inefficient public-sector agencies to lay off redundant workers and reduce bloat?
> Answer. None.
And keep in mind that academia might as well be considered public sector, but the idea transcends to most large-scale/loosely-planned cost cutting for efficiency gain schemes.
Your starting point is a lot on individual virtue, and bad apples crowding out the good. I don't disagree that some researches are better and some are worse, but I think that any "cure" is going to be gamed and worse than the disease.
> but the largest will be that as funding becomes a scarce resource, a higher curation of both researchers and topics will come into effect
Do you have any evidence of this? I disagree in the strongest of terms. By my reckoning, curation, preventation, and other "forsight-drive" work is persistently undervalue by our society and economy. The nastiness that will accompany shrinking funding as research groups fight for short-term survival will only make that worse.
> but the largest decrease in research will be those that are bullshit
Why would those who are the best at gaming incentives now loose that skill?
> It's been happening for years, centuries, as a basic economic cycle, and looks something like this: _/T\_/T\_/T\
So after centuries of Byzantine decline, there is new group of lean and mean Greek Constantinopolitan bureaucrats?
After the Spain grew rich on New World bullion and then poor with a dearth of industrialization there is a new generation of hyper-efficient factories putting Germany to shame?
I dunno what history you are reading, but the good actors never outlive the broken system and get the last laugh in mine.
> Your starting point is a lot on individual virtue, and bad apples crowding out the good. I don't disagree that some researches are better and some are worse, but I think that any "cure" is going to be gamed and worse than the disease.
I don't think that there will be any cure, just a change in available resources that changes the number of projects that get funding. Of course people will game it, but the people who dole out the cash will be more selective if there is less cash to dole.
>> but the largest will be that as funding becomes a scarce resource, a higher curation of both researchers and topics will come into effect
> Do you have any evidence of this? I disagree in the strongest of terms. By my reckoning, curation, preventation, and other "forsight-drive" work is persistently undervalue by our society and economy. The nastiness that will accompany shrinking funding as research groups fight for short-term survival will only make that worse.
There isn't any evidence of events that haven't occured yet. There is logical and historical support, as hundreds of years of economics have shown that market systems tend to optimize around the scarcest resources.
>> but the largest decrease in research will be those that are bullshit
> Why would those who are the best at gaming incentives now loose that skill?
They wouldn't lose that skill, but as there will be fewer research projects funded, more scrutiny will go into which projects get the funding. Sure, some will still game the system, but it will require more work, and more luck, so proportionally fewer will get funded.
>> It's been happening for years, centuries, as a basic economic cycle, and looks something like this: _/T\_/T\_/T\
> So after centuries of Byzantine decline, there is new group of lean and mean Greek Constantinopolitan bureaucrats?
Now you're just being silly... You're saying that the Institution of research will fall, rather than self correct? We'll just give up on scientific research and just... Hang out or something?
> I dunno what history you are reading, but the good actors never outlive the broken system and get the last laugh in mine.
It depends on the system. There have been years with drought, but people haven't given up on farming. Systems only fail when they are unsupported or unnecessary, and Institutional research is neither of those.
My wife, a health outcomes researcher, points out that the expectation is that every little publishable bit be kicked out the door for a pharma project.
Historically, they would have dropped a single summary result at the end of the effort.
Now, she's trying to "autotune" the neutral or even negative results. Nominally in the name of "transparency".
If this is true, the need is a research Adele to come along and shame the riff-raff with sheer natural brilliance.
Then almost always, a clone tool is used to repeat that one bar identically dozens of times.
So you lose most of the organic nature of music in a mechanical fashion.
Beside humans?
I don't understand the popularity of this argument. Whilst someone might (theoretically) have written the Great American Novel in his or her basement unbeknownst to pretty much anyone, a lot of art is fairly resource intensive and thus you can say with extremely high confidence that's its not being produced and the fault lies not with people not looking hard enough. Same with research and journalism.
On another note, why does everything have to be exciting? Small incremental improvements over the years is what I aim for.
E.g., the artist takes a big risk to produce a sound that diverges from “what works” just like a scientist takes a big risk to diverge from well-worn theory that has produced past publication fodder.
I think better structures are needed that actually incentivise good reviews. This would lead to better papers, too. There are multiple options for this. Maybe it is simply time to see reviews as publications themselves and say goodbye to flawed blinded review process. Another thing would be to take the publishers out of the game. Even IEEE and ACM have become largely a playing field for power politics between the US, China and Europe . I think we could do better and be more inclusive here. Publishing reviews and rejected work would further make meta studies much more reliable removing also bias towards positive results (who actually wants to hear about easily replicatable negative results at a conference)
It’s the failure to harness the hard work and brainpower of so many brilliant people towards efforts that would meaningfully improve the lives of others.
It’s just the nature of the game. You can’t just turn a dial and add more talent.
Although a bit off topic there are some authoritarian nation states who tried to do this by social engineering. Soviet Union made being an engineer a common place job title and China is trying to make tons of super smart kids at STEM. But we all know there is more to being smart than forcing kids to go down certain academic route and get top end scores (you know the whole creative side and capacity for original works to be created).
The democratic approach still seems to be the winner (not just due to brain). It’s as I mention always going to be a small minority who do great things. That doesn’t scale up artificially.
It’s true that we can’t turn a dial, but there must be a vast amount of human potential that goes unused every day, because people are born in impoverished countries, or suffer discrimination due to gender or race or something else, get sick with a preventable disease, or whatever other thing denies them the opportunity to make the most of themselves.
This is probably actually the norm in the world, not the exception.
So I think there’s rather a lot we could do as a species (or within individual countries) to give people opportunities and make sure potential for progress is realised
I'm not sure I agree there. It's notoriously hard to get/keep a research position, especially a) in academia and b) outside CS. There are a decent number of graduate student slots, some postdocs, and then you're thrown into the thunder dome.
The competition itself--and the resulting uncertainty--chases a lot of people into other careers. The people in my grad school cohort who "left" research were just as smart and often just as successful (in terms of papers, etc) as those that stayed.
It's not hard to imagine some policy tweaks that could have kept some of them in research.
There's research to support that actually:
https://www.insidehighered.com/news/2018/12/11/new-study-say...
Buried in that paper they basically show the hazard rate for exiting academics is sort of uncorrelated with metrics like citation rate etc.
Maybe after say, quantum computers are available, it takes around 25 years for a team of researchers to slowly make progress and integrate information until they find a killer application. Funding 25 teams of researchers may make them pump out papers 25x faster, but they aren't going to find the killer application in 1 year.
> so tax payers can sleep well that their money is paid for hard work.
It's a bit contradictory.
I won't go into scandalized rant because it's useless, and there's no point shouting against the universe from a bedroom chair. Still it's a bit disheartening. I wish the scientific world could reinvigorate and pump some spark back into the field.
-- paraphrased from Max Planck
There is another dimension to that though. It's the long tail. The internet has tremendously facilitated mixing existing content. That's creative too but not in the original sense of creating something new.
With detached knowledge before the internet age much less content has been produced and I would assume the novelty to mixed content ratio was very much in favour of new content instead of mixed content.
Internet technology brought us search engines but their search algorithmus favour those paying most instead of providing a service to discriminated remixed content from new content.
One answer to all these propositions is that the valuable stuff is still there, you just have an additional flood of mediocre boring predictable stuff. You can still find enough indie music and games and movies to spend all your free time on.
Similarly, if you know where to look and whose papers to watch out for, you can read more interesting stuff. The existence of more bullshit makes the filtering somewhat more effort, but it's bearable.
Most papers are never deeply read anyway, outside of the reviewers. Even citations don't mean someone really read the paper in depth.
So, most medíocre, boring papers get ignored.
It's an illusion to expect that every researcher can pump out multiple really interesting non-boring papers every year. So we pretend. We write, we cite, we present, overiflate, overpromise etc. It must look like there is steady hard noble work and toil. It's like a ritual. The scientists are writing all these papers so tax payers can sleep well that their money is paid for hard work.