(1) Many academics aren't aware non-academics read their papers at all: we work with other academics, go to conferences with other academics, and on the rare occasions we hear from readers, it's from other academics. Big exception: in some fields academia and industry have much more interaction, biomedical research (the subject of the linked article) being one of them. Extracting knowledge from that literature has a large number of practical and economic implications.
(2) There seems to be a perception that published papers are a repository of established or state-of-the-art knowledge. Perhaps they were meant to be that way, and perhaps more of them should be. But for many journals in many fields, publications are a form of moderated discussion. Reconstructing the state of knowledge from snippets of conversation is always going to be hard.
What can help make the literature more accessible? some of the forces are structural, some are due to current limitations of technology. But one thing that can help if you find the results of a paper interesting and are able to track down the authors, write them. People like hearing their work is noticed, and they like talking to people about things they're interested in.
Another is to make constructive suggestions (or even pitch in to improve code where it's open source & available). Between teaching, advising, committee work, etc (not to mention family), most of us have to prioritize, and as much as I'd like to clean up old code for release in the hopes someone finds it useful, it isn't going to get my grad students out the door with a degree or a job -- I'm generally spending more time on their research problems these days than my own. But if I know there's interest / use I might prioritize time a little differently.
Review articles, sometimes called surveys.
I've always thought that new PhDs would be excellent authors for those, having digested lots literature for their dissertation.
A well written PhD or MSc thesis is often the best way into a new field, ime. If the committee is good on this aspect they'll insist you've put enough detail in for someone to follow along mostly self contained.
The text changes to fit the audience, and the knowledge becomes more accepted (and or fundamental) further down the line.
Is this field specific? I have read survey articles in math and biology, and was told by some of my profs that they use these articles as an introduction to a new field.
A quick Google search seems to show these exist in CS (along with tutorial papers), physics, and chemistry but I'm having a little difficulty finding statistics survey papers (survey methods come up instead).
Is the problem that there aren't enough of them or they are behind paywalls?
Please suggest others if you find them.
Annual Reviews has a bunch of journals for surveys of various fields. Most of them are paywalled, but there's ways around that.
Publishing code and data associated with figures. A significant amount of the confusion about the literature comes from the difference between the documentation laid out in a paper and the reality of the actual implementation. Plus any peer review of the actual code used is checking the reality of the paper, rather than a description of what we think reality is.
Making this part of the publication process is vital, because there is such a high barrier to actually requesting this information later.
Full disclosure: your username is very easy to Google, and I should tell you we work on closely related topics. My opinion is shaped in part by several of the papers you've cited. I'm happy to disclose my identity and continue the conversation in private.
You wrote: "(1) Many academics aren't aware non-academics read their papers at all"
What, then, is the point of applied mathematics? Please know that I don't mean that in a dismissive way. I think there are a few reasonable answers. Chief among them is the belief that exploratory research is important in its own right and does not need to have an immediate non-academic use, as embodied by this quote of Hadamard:
“Practical application is found by not looking for it, and one can say that the whole progress of civilization rests on that principle.”
I believe the above quote as far as it concerns mathematics. But how do we get from there to the applied part? I have an internal dissonance about this that goes deeper than just semantics. Before I started my PhD, I had some vague belief that after writing up some research with an algorithm in it, you'd put it on the arxiv, and from there someone might one day need something like that, code it up, and use it. If I could put in a basic working implementation that was even better.
All the evidence I've seen so far tells me this is not so. The truth is no one is going to take the time to code up your algorithm, because no one has dozens of hours to spend understanding your paper, developing an algorithm suitable for an industrial problem, often just to get improvement on a niche subset of cases. I've been wondering how to estimate the number of algorithms described on the arxiv that are ever implemented and used in a non-academic setting -- my bet is (outside of ML), less than 1%.
I've heard many times that sophisticated higher-order methods for PDEs (finite element / volume, Galerkin, ...) are used in aeronautics, to determine the wind shape over an airplane wing. I've found out from talking to people in the industry at companies like Bombardier that for the most part they do second order finite difference like the rest of us. Why? Because you can code it up in an afternoon, whereas writing the more sophisticated methods can take weeks or months. As academics, we think that the theoretical work is the really hard part, and we neglect the human cost of writing and maintaining algorithms. We have it backwards: academics are (relatively) cheap; code (and changing code) is expensive. (Of course, I make these comments assuming a certain scale. We can come back to this.)
I think the fundamental issue is that I know few applied mathematicians who start with a problem and seek out a solution. Most often, you finish your (applied) math PhD armed with some machinery. If you want to get a professorship and you've done well, you typically turn the crank of your particular machine better and faster than most. In applied math we can say our model is motivated by some problem in the sciences/economics/whatever, but in my experience that just lets us erect a straw person (create a problem) and tear it down (solve the problem) using the machinery that only we have mastered. Just because a problem is hard doesn't make it important.
What to do, then? How do you work on "consequential" problems?
To be pithy about it, I've found it useful to think in terms of $ rather than h-index. In many cases, a consequential problem is one that, if you solve, you can monetize. You could frame this as asking what kind of mathematics could enable new technologies. In my experience it is very difficult to write down a mathematical question that, if answered, can lead to new technology. But if you manage to find such a question -- and it is possible -- it can be a goldmine.
I have more to say -- especially about how mathematicians need to get a reality check on the importance of hardware and its relevance in stochastic algorithms research -- but this is long enough as it is, and I don't want to just be a crazy person rambling in the corner. I'd be very curious to hear your thoughts.
There are papers that are well-written and useful, but there are at least as many that are just drivel (I probably contributed to both kinds).
Unfortunately, the prevailing attitude is that outside people will not understand our stuff anyway, so we often make no effort to make papers understandable, or to publish data. (There is a lot of great outreach and science communication, but not so much for students or researchers from other fields who want to follow the technical details.)
Your results replicate, or they don't. Your calculations, equations, and models predict experiment. Or they don't.
Writing papers about it and getting the feedback of "peers" is nothing more than an old fashioned circle jerk for padding resumes, CVs, and persuading other people in that academic hierarchy that you deserve funding. It is a game that is divorced from actually learning, researching, understanding, measuring, and predicting the world.
Not saying incentives are perfectly aligned -- many citations are superficial ("this topic was studied before"), and papers count for a lot even if they're never cited, etc
In opensource the attitude is "See bug? Send a PR!"
Whereas academic papers are like publishing software into a blockchain (and not source but binaries, i.e. PDFs full of shortcuts): you don't want for people to easily find bugs and contribute fixes, so you handwave a lot so that no one can reproduce your exact thing.
The difficulty in such a thing would be the journals and database companies are holding on to their exclusivity and profit motives with an iron fist, so unless you want to get sued into oblivion, you'd have to stick with open source or accessible articles, so you'd need to either specialize in disciplines that have moved away from closed-source enough that the tool wouldn't have massive holes in it.
Also determining which new references and reviews have relevance (like if anybody can comment with new references, who goes through to check they're actually relevant or say what the person says they say?), preventing academics/administrators from gaming the system if it DOES get popular, etc. In open source, this is crowd-sourced, but for some academic fields the number of people who are qualified to speak on a matter is extremely small.
/academic librarian thoughts
It'd be viable for fields that don't use/rely on for-profit or closed journals, but I don't know if the money to run it would be there, especially since the odds of the big Schol Comm players suing is still there, because it'd be worth it to ruin the tool/effort before it can challenge them.
Building this would be my dream job, but hahaha no.
I think there was one pull request total?
The juice just didn't end up being worth the squeeze.
Seriously though, you're totally right. I got very dissatisfied with science when I realized that many people were effectively publishing unreproducible crap created by terrible code. Fortunately, more and more people are learning how to recognize the crap.
It doesn't have to be this way. Here's the process I use in my lab:
1. Every paper that makes a claim of any kind based on code contains a link to a public Git repo.
2. The paper contains the Git hash identifying the exact commit used to justify the claim. Copy-paste it from the paper into your checkout.
3. The repo may have moved on with fixes and improvements, you can have those too.
If you are using version control already, it's not much work to do this. Of course, you have to be committed to making your code public.
If you think the entire field of academia doesn't achieve any purpose, you may want to reconsider your position. Most likely, almost everything that you do today on a computer was an academic paper. Yes, it was without code and data. Yet, it was not unusable and achieved more than enough purpose.
The average comment on HN on academia comes from a mindset where everyone wants a product. The purpose of a paper is NOT to release a software or a product. But, to test an idea under some assumptions. That's what all research does at its core - formulate a hypothesis, design an experiment to test the hypothesis and report the results and implications. Are all research papers perfect"? No. Are all of them usable? No.
Your use case - sound synthesis for a specific instrument - may not be a scientific challenge. It is however an engineering challenge and hence, you found a better answer amongst hobbyists and tinkerers. Now, try looking for the a vaccine for Covid - and guess where you'd find that answer? In decades of research on mRNA with repeated failures, papers that couldn't be replicated, unavailability of "code" and samples with verbal descriptions skipping crucial details.
"The thing is, I don’t care if something has a thousand retweets, what I care about is if it has two or three independent confirmations from economically dis-aligned actors. This is the same as academia, by the way, everybody’s optimizing citations. What you actually want to optimize is independent replication. That’s what true science is. It’s not peer review. It is physical tests."
Others have commented as well but I will reinforce: their output is basically unusable for you for the purpose you want to put it to.
Which is fair, but you should also recognize that you are not the audience of the papers and for good or for ill the system is not set up to help you with this.
I agree that these days the tooling makes it much easier to distribute code & data somehow to match up, but there is also a cost/incentive mismatch. Basically to do a decent release of what you are working on and worse, potentially support it, costs time but has no real career value (yet). Which means it's mostly only done by people who are philosophically convinced of its value.
I think this will change over time, at least in some areas, but it won't be quick.