News for hackers. Stuff that matters.
Re:Why isn't preprint review being adopted? by geysersam on Sunday March 24, 2024 @05:21PM (#39810636)
Or, research just gets published online free of charge for everyone to access, and important work will prove itself over time by being discovered, discussed and by becoming influential.

If anyone wants to change something about an article (the writing, the structure of the paper, or anything else a reviewer might want to edit) they can just do it and publish a new version. If people like the new version better, good, if they don't they can read the old version.

Peer review as a filter for publishing is terrible in a time when making a few megabytes of text and images accessible is literally free. If anyone wants to run a content aggregator (aka a Journal) they just do it. If they want to change something about the article before it's approved for the aggregator they can contact the authors or ask someone to review it or whatever.

Just make it accessible.

Re:Why isn't preprint review being adopted? by ska on Sunday March 24, 2024 @06:23PM (#39811048)
> Just make it accessible.

We already have that system, it's called the internet. Nothing stops you or I from putting our ideas online for all to read, comment on, update, etc.

The role of the publishers, flawed as it is, has little to do with the physical cost of producing or providing an article, and is filling (one can argue badly) a role in curation and archival that is clearly needed. Any proposal to change the system really has to address how those roles are met, hopefully cheaper than currently but definitely not more expensive because mostly people don't get paid (in $ or career or anything) now - or has to provide a funding source for it.

I don't really see how your outlined scenario addresses that, at least not in a way that's functionally different than today. Can you expand?

Re:Why isn't preprint review being adopted? by paulmd on Sunday March 24, 2024 @10:12PM (#39812303)
> We already have that system, it's called the internet. Nothing stops you or I from putting our ideas online for all to read, comment on, update, etc.

Are you asking how arxiv is different from blogspot?

Re:Why isn't preprint review being adopted? by ska on Sunday March 24, 2024 @10:32PM (#39812397)
No, although the mechanism for hosting the content aren’t that important.

Preprint servers are very useful but haven’t replaced journals for good reasons.

Re:Why isn't preprint review being adopted? by staunton on Monday March 25, 2024 @10:26AM (#39816763)
What are those reasons? The only thing I see that journals do which preprint servers couldn't easily take over is prestige.
Re:Why isn't preprint review being adopted? by ska on Monday March 25, 2024 @03:09PM (#39820151)
You have the causality wrong. Prestige comes to journals by doing a good (or at least, better than peers) job of being a journal, which is providing a necessary function to the academic research process. If you want to improve on that system, you have to improve on those functions, or reduce the reliance on them by providing something better.

Put it another way, if you can design a system with a better ROC curve for classifying research, with a better TP rate for good papers, and have it cost less in real terms that current academic papers, then you are on to something. If all you've got is "papers should be free" or "it's too hard to access publishing from the outside" what you have are complaints, not solutions.

Re:Why isn't preprint review being adopted? by staunton on Monday March 25, 2024 @07:45PM (#39822581)
Like I said, it's not clear to me what exactly established journals have been doing "better" historically than a preprint server could. You say they are better at "being a journal" -- OK. They are established, well connected (to science communities, industry, journalism, funding agencies, etc.) and have been maintaining a reputation for a long time, usually much longer than preprint servers exist. That's basically prestige, which isn't nothing (I didn't claim prestige is nothing). However, this doesn't demonstrate journals do anything relevant to the advancement of science fundamentally better right now.

What I "have" is that

1. It's not obvious that a journal is fundamentally better at organizing unpaid voluntary reviewers compared to a preprint server.

2. Scientific publishing has insanely high profit margins. How come? My theory is that they are selling prestige first and foremost, i.e., a luxury good, (to scientists, universities and funding agencies simultaneously) and purchasing decisions there are made by people who are spending public money, not their own. Both of these points (luxury good, public spending) seem like strong contributors to high margins. The public is paying for the research and for access to articles, while journals nowadays on first glance seem to only provide a little bit of web hosting, a little bit of reviewer coordination and a little bit of manuscript editing.

3. It's not obvious that the submission and peer review systems we have now (in journals) is worth the time and effort. The role of peer review is misrepresented in journalism and the expectations are not met. If one could separate publication ("preprint") on one side, and, on the other side, review and "being featured by important outlets or institutions", authors could save a lot of time time (that could be used for more research). Others would have access to interesting results earlier and be able to build on top of them. Next, in a separate process some institution could select important works, scrutinize and review them, perhaps paying experts to do so, and perhaps replicate where appropriate.

The issue with this is that academics need the prestige provided by journals for career advancement, universities need the prestige to justify their spending to funding agencies and politicians, and funding agencies likewise need the prestige to justify their spending to politicians. The "replication crisis" and the like indicate that this prestige is overvalued. The hope is, economically speaking, that the market for "academic prestige" can either be disrupted, or the price the public has to pay can be lowered "through competition". It's interesting what that might look like. Preprint servers, open data and more direct science communication seem like steps in the right direction.

Re:Why isn't preprint review being adopted? by rhelz on Monday March 25, 2024 @09:23PM (#39823341)
Your concerns are valid, and I think that my proposal of paying reviewers and replicators addresses them all.

> It's not obvious that a journal is fundamentally better at organizing unpaid voluntary reviewers compared to a preprint server.

So, let's not rely on unpaid, voluntary labor. Pay them.

> they are selling prestige first and foremost,

Yes. So give them a better business model--if they can make money reviewing papers, they won't have to create artificial value by creating artificial scarcity.

> The role of peer review is misrepresented in journalism and the expectations are not met.

If you pay somebody, you can specify the expectations you think should be met. If they don't meet those expectations, they don't get paid.

> this prestige is overvalued.

The prestige is not overvalued--it is just too easily obtainable by fraud. Something has got to be done.

> Preprint servers, open data and more direct science communication seem like steps in the right direction.

They are vanity presses.

And they don't even do what you think they are doing. Today, the problem isn't too little information, it's too much misinformation. LLMs can chug out papers by the millions. Is a search engine going to help you cut through that and find what you are looking for? What if 2 million papers which match your search criteria? You gunna read through them all, trying to find the 5 papers which were actually written by a real scientist?

Are you even going to see them? Is the search engine going to do a better job than peer review of presenting you the papers you actually want to read?

Re:Why isn't preprint review being adopted? by staunton on Tuesday March 26, 2024 @03:49AM (#39825150)
> my proposal of paying reviewers and replicators addresses [all concerns]

Your other comment didn't say who might be paying reviewers. Journals clearly won't (why should they, they have grrat profits in the current system and will fight tooth and nail to delay any changes whatever). Universities and even funding agencies cannot (conflict of interest).

> Is the search engine going to do a better job than peer review of presenting you the papers you actually want to read?

I do actually expect to see that happen.

Re:Why isn't preprint review being adopted? by rhelz on Tuesday March 26, 2024 @10:04AM (#39827964)
> Your other comment didn't say who might be paying reviewers.

In the parent comment to this thread, I talk about this. My proposal is that when a researcher writes up the grant to get their research funded, they should estimate how much it would cost to pay reviewers and replicators, and include those figures in the cost.

If the researcher gets the grant approved, then the funding agency will put the money for peer review/replication into escrow. When the research is finished and the investigators have written up a paper to describe their methods and results, the money in escrow is disbursed to the reviewers and replicators.

If reviewers agree its good, and if it replicates, then the paper is published. If not, well we just dodged a bullet.

> I do actually expect to see that happen.

Are search engines getting better or worse for you? It was a lot easier getting the right paper from a search engine 10 years ago. Now, you just get half a page of irrelevant ads, and another half page of links boosted by payola.

Just imagine what it will be like when there are literally MILLIONS of bad papers for each good paper. Then Billions. There is no finite limit to the amount of bullshit that LLMs can--and therefore will--output.

Re:Why isn't preprint review being adopted? by ska on Monday March 25, 2024 @09:55PM (#39823529)
I'm clearly not articulating my point well. Obviously the idea is to "disrupt" the academic publication and review process, but this discussion seems to be focusing on the probably the easiest part - making and hosting the documents.

> Next, in a separate process some institution could select important works, scrutinize and review them

This is basically what happens now. Pre-prints are for things that aren't necessarily ready yet (hence the "pre") but cooked enough to review and discuss and build on. The formal publication process takes some percentage of them (depending on server, could be quite small) and works through a publication process.

Currently that is mostly done by for-profit journals organizing the work. If you want to propose.

So what you are suggesting is that we do away with that (fine!) and replace it with --- something handwavy (not fine). There has to be some real proposed mechanism of organizing the work that needs to be done that a) doesnt' waste even more time of the limited pool of people who can and will do a reasonable job of reviewing, even worse editing, does at least as good a job filtering out the large amount of noise to find signal, and is at least as robust against manipulation.

For what it's worth, many of your arguments about the lack of efficacy of the system or other flaws don't seem to me to capture how much worse it could be. Best not lose track of that in trying to make it better....

Re:Why isn't preprint review being adopted? by staunton on Tuesday March 26, 2024 @04:17AM (#39825300)
> So what you are suggesting is that we do away with that (fine!) and replace it with --- something handwavy (not fine).

I wasn't really trying to suggest any concrete system to replace the current one. Neither would I be able to do so nor would it really matter since such a system couldn't be implemented in a top-down fashion. I was pondering how things are and why, which is hard enough, as well as what trends I see positively (which are simultaneously actionable recommendations for both funding agencies and scientists).

> many of your arguments about the lack of efficacy of the system or other flaws don't seem to me to capture how much worse it could be

Sure, I think science as a whole has never been more productive. Many trends also look positive: besides what I named above, there is also increased industry collaboration for applied research, increased funding overall, etc. The main challenge will be the price of creating fraudulent submissions going down and hacking the system becoming more prevalent. I think the only way to address this is to significantly reduce the "perceived authority" of any work that comes from using a LaTeX template, as well as authority that comes with the label "peer reviewed".

Re:Why isn't preprint review being adopted? by ska on Tuesday March 26, 2024 @12:42PM (#39829938)
> I think the only way to address this is to significantly reduce the "perceived authority" of any work that comes from using a LaTeX template, as well as authority that comes with the label "peer reviewed".

Opening up access unavoidably makes the signal to noise problem worse, not just for the reasons you note (fraud, exploits) but also average quality drops. Whatever changes are made, will need a more effective filter, not less effective.

Re:Why isn't preprint review being adopted? by geysersam on Monday March 25, 2024 @02:31AM (#39813264)
That's just not true. Most publicly fundes research is hidden behind paywalls.
Re:Why isn't preprint review being adopted? by ska on Monday March 25, 2024 @02:52PM (#39819964)
No it's exactly true. You can write up anything you want and put it on a site. The post I was replying to was suggesting an open access system (both read and write) for exchanging ideas. This exists.

What it doesn't do is effectively replace the non-open system for access to academic journals. I have a lot of sympathy for open (read) access to research, particularly publicly funded. It just isn't sensible to wave a wand and say "all papers are free to read now" without some plan for the other parts of the system and the ecosystem (academic research) that relies on it.

Re:Why isn't preprint review being adopted? by geysersam on Monday March 25, 2024 @07:07PM (#39822276)
Why do you think that peer review needs the journals to function?
Re:Why isn't preprint review being adopted? by ska on Monday March 25, 2024 @09:45PM (#39823460)
I didn't say that. It needs something. Handwaving about an emergent community isn't useful - moving from todays system to something else needs something concrete.
Re:Why isn't preprint review being adopted? by rhelz on Sunday March 24, 2024 @06:13PM (#39810996)
Whether or not your papers pass peer review—-and which journals it is published in—are important criteria for hiring, tenure, whether your grants are funded, etc.

If you get rid of peer review, it’s not science. It’s just a vanity press.

Re:Why isn't preprint review being adopted? by bumby on Sunday March 24, 2024 @07:37PM (#39811517)
>If you get rid of peer review, it’s not science. It’s just a vanity press.

If you believe replicability is central to science, the current paradigm doesn't necessarily converge on science either. And when people are graded on how many publications they garner, it borders on turning publication into a symbol of status rather than one of science.

Re:Why isn't preprint review being adopted? by rhelz on Sunday March 24, 2024 @10:38PM (#39812419)
> If you believe replicability is central to science.

I do believe that, but it doesn’t matter what anybody believes, replicatable experiments and results, which peers can review and agree on, are the soul of science.

Without that it’s not science, it’s just creative writing.

Re:Why isn't preprint review being adopted? by bumby on Sunday March 24, 2024 @11:21PM (#39812590)
Then don’t you think the first part of that process is explaining the methods and results of your experiment? That’s precisely what the current situation does. What’s lacking is the replication incentives.
Re:Why isn't preprint review being adopted? by rhelz on Sunday March 24, 2024 @11:35PM (#39812655)
I propose to supply those incentives—-pay for good reviewer and replicators.

If you want something done, you gotta pay for it. We can’t just rely on volunteers.

Re:Why isn't preprint review being adopted? by bumby on Monday March 25, 2024 @11:27AM (#39817522)
I'm on board with that idea, as long as we can also provide guardrails against the perverse incentives of paying for them. E.g., we need to avoid frivolous reviews/replication as well as something evolving into a "pay for a good review" service.
Re:Why isn't preprint review being adopted? by rhelz on Monday March 25, 2024 @07:57PM (#39822678)
Well, if you are paying somebody to do something, you gain a lot of leverage:

1. You can negotiate a due date. No more waiting for years before the journal's reviewers actually review your paper.

2. You can negotiate a set of deliverables. You can specify that they can't just say "this sux"; they have to show the lines where the big hairy proof are wrong, or if its an algorithm, they have to actually code it up and run it themselves before they say it doesn't work.

3. You can more reliably attract good reviewers. If you aren't begging for people to volunteer, but you are paying good money, you can be a lot pickier about who you hire.

I mean, I've been a consultant: what are the guardrails that I won't rip-off my clients? I don't want to ruin my reputation, I want repeat business, and I want to be able to charge a high hourly rate because I deliver a premium product.

Same guardrails would apply to peer reviewers and to reproducers.

Re:Why isn't preprint review being adopted? by bumby on Monday March 25, 2024 @09:30PM (#39823385)
Sure, but I’m poking at the bad leverage you can also wield.

1) you can create undue schedule pressure that results in a cursory review that may not catch the more nuanced problems in your investigation.

2) you can be more belligerent about not sharing data. If they want to get, they won’t argue.

3) you can pay for reviewers who you know will give a positive review. Without guards against this, it’s almost a certainty that the glut of PhDs will result in some treating it like a scam de hustle where it’s more about the economics than the science.

Some consultants are well known to play the game where they tell clients what they want to hear rather than what they need to hear. I don’t think consultancy is a good model for this.

Re:Why isn't preprint review being adopted? by rhelz on Tuesday March 26, 2024 @10:14AM (#39828085)
#1 isn't an issue unique to paying peer reviewers. We've learned how to negotiate such hazards.

#2 Seems like a team who wants their paper published would be super-helpful to the reviewers and replicators....why wouldn't they be maximally motivated to help them to by sharing data, techniques, etc...and writing their paper so that its easy for reviews and replicators to do their jobs.

#3 The authors of a paper don't get to choose who their reviewers are!!

> consultants...play the game

And yet we have millions of clients hiring millions of consultants, and somehow they are able to make it work....yeah, all these issues can arise in other contexts, we know how to deal with them.

Re:Why isn't preprint review being adopted? by bumby on Tuesday March 26, 2024 @04:28PM (#39832464)
You are right that #1 isn't unique. But I think you're wrong that we've got the issue easily solved because it's rooted in human psychology. Just look at the last few years of Boeing headlines and tell me you still think schedule pressure in a competitive environment is a solved problem.

Your response to #2 assumes the researcher wants to create the most transparent and highest-quality paper. Because of perverse incentives, I don't think this is the case. Many times researchers just want a publication because that gets them the career status they're after.

Good point on #3, but it still leaves the question about the tradeoff between quantity and quality. I can surely churn out many more reviews of questionable quality than I can a single, well-researched and thoughtful review. The quantity vs. quality tradeoff is really what is at the heart of that point.

>And yet we have millions of clients hiring millions of consultants

The existence of that market doesn't mean the market does what you're claiming. Many times, consultancy is a mechanism to pay for plausible deniability rather than a novel solution.

Re:Why isn't preprint review being adopted? by rhelz on Tuesday March 26, 2024 @06:00PM (#39833355)
re #1: Yeah, bad apples will be bad apples, but that doesn't stop us from hiring people to build us airplanes and run aerospace companies. Right now we are assuming that humans are so angelic they will give us quality reviews for free.

re#2: Under my proposal, an researchers in an independent lab would have to read a paper to see how to design and conduct an experiment to replicate the results. And if it didn't reproduce, they don't get their paper published.

Given the stakes, don't you think researchers would exert every effort to make their paper as transparent and as easy-to-read, as possible? How carefully would they describe their experiment if they knew somebody was going to take their description and use it to check their work?

Re #3: Yeah, but again that's not a problem specific to my proposal. The same risk hangs over every employer-employee relationship.

Re:Why isn't preprint review being adopted? by bumby on Wednesday March 27, 2024 @10:46AM (#39839922)
I think the idea of requiring the review process to require replication would potentially be a good approach, given we're aware of the downsides. For example, I've worked in labs with sensitive data, or with proprietary processes that they would not want to share. This would mean the advocated process would result in a lot less sharing of methods. Maybe there's a chance there could be vetted independent labs that meet stringent security requirements, but that adds another layer of bureaucracy which could, again, result in less sharing of information. There's a balancing act to be considered, and I agree that we are probably too far on the one side of that equation currently.

Most of your rebuttals seem to hinge on "yeah, but that problem isn't unique to publishing." That is a kind of side-stepping that misses the point. The point is we need to create a system that mitigates those downsides, not ignore them. I I don't think a store manager would be okay saying, "Well, people steal from all kinds of stores, so we don't need to try to minimize theft." They recognize stealing is a natural outcome given human tendencies and create a system to minimize it within reasonable boundaries.

Re:Why isn't preprint review being adopted? by rhelz on Wednesday March 27, 2024 @04:20PM (#39844112)
> Most of your rebuttals seem to hinge on "yeah, but that problem isn't unique to publishing." That is a kind of side-stepping that misses the point.

If there is a specific objection you'd like to revisit, I'd be happy to discuss it. But I wouldn't self-describe what I'm doing as "sidestepping"--I'd say its avoiding bike shedding and keeping the conversation focused.

I mean, it's a pretty facile objection to say some variation on "but if we pay them how do we know we'll get our money's worth?" when we pay for goods and services all the time with very high confidence that we'll get what we pay for.

Surely, there's plenty of considerations to discuss, and I've tried to squarely address all objections which are specific to this proposal. But how to hire and use consultants, or how to ensure you get what you contracted for, are largely solved problems, and off-topic.

> This would mean the advocated process would result in a lot less sharing of methods.

I don't think my proposal would even apply to internal R&D groups who wanted to keep things proprietary. I mean, I can certainly understand wanting to reserve some methods or data as being proprietary. But choosing to do that is, ipso facto, not sharing them. How would paying reviewers and replicators for their time cause any less sharing to happen?

I mean, if your paper doesn't describe the experiments you performed in enough detail to allow other groups to replicate it, its not a scientific paper to begin with. It's either a press release, or a whitepaper, or some other form of creative writing, and publishing it is either public relations, or advertisement--not science.

Which is not to say that it's immoral or useless, or to denigrate it in any way. Not everything we do has to be science. My proposal is just for scientists communicating scientific results with other scientists. Maybe I'm missing something, but I don't see how it would inhibit the kinds of practices you are describing in any way.

It would make it harder for people to claim their "results" are scientific, but are not. It would be a big obstacle to publishing fraudulent papers in scientific journals. It would make it harder for somebody to claim the mantel of "science" to give credibility to their claims. But I really don't see how paying reviewers and replicators would stop anybody from sharing as much or as little as they wanted to.

Re:Why isn't preprint review being adopted? by bumby on Thursday March 28, 2024 @04:22PM (#39856840)
Apologies, but when the central claim is about mitigating downsides of adding money into a system and you acknowledge the potential for downsides exists but fail to provide any mitigation, it is sidestepping the main focus of the discussion.

I also think there is a misunderstanding when you’re talking about internal R&D. The situation I’m talking about isn’t where someone wants to protect a proprietary method, but rather proprietary data. I could have sensitive information that I don’t want to share, but also recognize a method I’ve developed is useful to others. The harder you make it to share that method (by requiring me to sanitize all the data to make it non-sensitive) the less likely I’m going to share it. When things like security or law come into play, the easiest path is always “no.”

>If there is a specific objection you'd like to revisit

Take the fact that whenever you inject pay into a system, it tends to pervert that system away from the original goal and into a goal of maximizing pay. You acknowledge that but just say it isn't unique. I agree it's not unique, but what I'm after is how do you propose to mitigate it (assuming your goal isn't to simply maximize pay, but rather provide some balance of quality, pay, and quantity). What guardrails do you put in place? Maximum on the number of reviews per quarter? That might limit those reviewers who can crank out many quality reviews. Do you instead provide a framework for reviewing the reviews for quality? That adds another layer of bureaucracy to an already bureaucratic system. Do you implement reviewer scorecards? A decaying rate of pay for each review?

And on and on. Again, the intent wasn't to imply these are unique problems but to probe for good fixes. Those aspects you say are digressions (consultancy etc) are topics you brought to the discussion, seemingly to address the mitigation question without actually providing a specific response. Doing "whatever they do elsewhere" isn't really an answer.

Re:Why isn't preprint review being adopted? by geysersam on Monday March 25, 2024 @02:36AM (#39813285)
And the current system is a holy grail that can never change? You can still based your hiring based on reviews made by peers