News for hackers. Stuff that matters.
Re:Fake scientific papers are alarmingly common by Turukawa on Saturday May 13, 2023 @11:23AM (#35929254)
The researchers in this paper use an astonishingly biased "fake paper detector", requiring only two conditions to be met for any paper to be considered "fake":

1. Use a non-institutional email address, or have a hospital affiliation, 2. Have no international co-authors.

And they acknowledge 86% sensitivity and 44% specificity. It's a coin-toss which biases massively against research from outside the US and Western Europe.

This "paper" is bigoted nonsense.

https://fediscience.org/@ct_bergstrom/110357278154604907

Re:Fake scientific papers are alarmingly common by FabHK on Saturday May 13, 2023 @11:36AM (#35929389)
No. They use 400 known fakes and 400 matched (presumed) non-fakes to estimate the sensitivity and specificity of their indicator, then apply that indicator to the full universe, then employ the estimated sensitivity and specificity to the obtained measurement to estimate the approximate actual rate of false papers.

If you know the true prevalence of a disease in a population, and the sensitivity and specificity of your test, you can predict how many positive measurements you obtain. Vice versa, from the (flawed raw) measurement, given sensitivity and specificity, you can estimate the true prevalence.

Furthermore, they’re explicitly saying that “red flagging” by their simple indicator doesn’t mean that the paper is fake, but that it merits higher scrutiny.

ETA: I mean, it could still all be bullshit (by virtue of some bias or so), but you’ll need to argue a bit harder to establish that.

Re:Fake scientific papers are alarmingly common by newswasboring on Saturday May 13, 2023 @11:53AM (#35929577)
> Furthermore, they’re explicitly saying that “red flagging” by their simple indicator doesn’t mean that the paper is fake, but that it merits higher scrutiny.

Then they and science should change their sensationalist headline. It's ironic that a paper about fakeness of something uses a borderline misleading title.

Re:Fake scientific papers are alarmingly common by danhau on Saturday May 13, 2023 @12:14PM (#35929800)
You’re not wrong, but it is everyone’s own responsibility to read the article and not just the headline.
Re:Fake scientific papers are alarmingly common by newswasboring on Saturday May 13, 2023 @12:22PM (#35929866)
So it's ok to lie in a portion of your work? Where do you draw the line? I draw it when someone starts communicating. Being wrong is ok, being deceitful isn't.
Re:Fake scientific papers are alarmingly common by caddemon on Saturday May 13, 2023 @03:45PM (#35931994)
Is this headline really deceitful though? Certainly the research is flawed, but the statement "[bad thing] is alarmingly common" is basically just a subjective statement that lets you know what position the author is going to argue.
Re:Fake scientific papers are alarmingly common by newswasboring on Sunday May 14, 2023 @12:08AM (#35935243)
I will never understand why everyone bends over backwards to justify lazy af journalism. This a magazine which is supposed to do scientific journalism, yet it didn't even mention the points that readers in HN comments were able to figure out on a cursory look. Peer review isn't just the 3 reviewers who accept or reject something in a journal. It's everyone in the scientific community.
Re:Fake scientific papers are alarmingly common by ouid on Saturday May 13, 2023 @12:37PM (#35930022)
Responsibility is not conserved in a robust system. This is true and it is also the journal's responsibility to not mislead.
Re:Fake scientific papers are alarmingly common by Retric on Saturday May 13, 2023 @03:33PM (#35931878)
Expecting people to read every single article posted to HN is unrealistic.

Simply reading a title and on a topic you don’t find interesting then gives people the wrong impression.

Re:Fake scientific papers are alarmingly common by tgv on Saturday May 13, 2023 @11:56AM (#35929615)
Their baseline had better be representative.
Re:Fake scientific papers are alarmingly common by marcosdumay on Saturday May 13, 2023 @12:27PM (#35929922)
So, in other words, the signal they get from it is around 70% of the noise, but it's ok because you can indeed do that with good enough statistics?

They better have a flawless methodology, because any tiny problem is enough to ruin their analysis. And well, just flagging almost any paper not from the EU or US as fraud doesn't usually come together with a flawless methodology.

Re:Fake scientific papers are alarmingly common by steppi on Saturday May 13, 2023 @12:34PM (#35930003)
I've worked on research estimating prevalence from imperfect tests, and something that concerns me about this study is that they aren't showing the error bars for their estimates. Typically, you would report a confidence interval for prevalence rather than just a point estimate, and the confidence intervals can often be fairly wide. There's two sources of uncertainty here, the assumed probabilistic nature of the diagnostic test, and uncertainty in our estimates of the sensitivity and specificity.

I think this paper by Peter J Diggle [0], gives a solid methodology. Instead of treating sensitivity and specificity as fixed values using sample estimates, you can model them as each having a beta distribution. In this case these beta distributions can be found using a Bayesian treatment of Bernoulli trials.

[0] https://www.hindawi.com/journals/eri/2011/608719/

Re:Fake scientific papers are alarmingly common by steppi on Saturday May 13, 2023 @03:31PM (#35931861)
Amazing. Reading more carefully, as FabHK pointed out above, they aren't even applying the obvious correction. They're just reporting the positive rate of the imperfect test. I've implemented Diggle's method [0]. When I have time, I'll see if they've provided enough data to do a proper analysis, and maybe write a blog post about it or something.

[0] https://github.com/indralab/opaque/blob/761572ed1b0d601271f0...

Re:Fake scientific papers are alarmingly common by robocat on Saturday May 13, 2023 @04:28PM (#35932370)
> they aren't showing the error bars

Perhaps any paper without error bars should be tagged as a fake paper.

This one would have sneaked past though: https://retractionwatch.com/2022/12/05/a-paper-used-capital-...

Re:Fake scientific papers are alarmingly common by Retric on Saturday May 13, 2023 @03:53PM (#35932069)
You can’t directly calculate both sensitivity and specificity using equal numbers of positives and negatives groups unless the actual population has that ratio.

A completely random test given equal populations results in 50% accuracy and 50% specificity. Things don’t look nearly as good if only 1% of the actual population has the condition.

Re:Fake scientific papers are alarmingly common by ekianjo on Saturday May 13, 2023 @11:54AM (#35929587)
> The researchers in this paper use an astonishingly biased "fake paper detector"

I havent looked at the details here, but if you make a prediction model and if that prediction model is robust enough to explain with great accuracy something with 2 or 3 variables, it's not going to be "biased", it's just going to be robust and right more often than not using only these few variables (as long as the training data was broad enough).

Re:Fake scientific papers are alarmingly common by GalenErso on Saturday May 13, 2023 @01:54PM (#35930840)
Why? Why can't scientists from outside the US and Western Europe seek international co-authors, like everyone else?
Re:Fake scientific papers are alarmingly common by detaro on Saturday May 13, 2023 @01:56PM (#35930868)
Why don't you consider having to do that a bias against them?
Re:Fake scientific papers are alarmingly common by jknoepfler on Saturday May 13, 2023 @04:31PM (#35932395)
So reading the actual article and the study they cite (https://www.medrxiv.org/content/10.1101/2023.05.06.23289563v...), there's a pretty compelling story being told.

Paper mills are a $3-4 billion dollar industry that is growing rapidly. That money isn't coming from nowhere. There are a lot of fake papers, and the fake paper industry is growing steadily.

So then the question becomes "where are those fake papers being published, and by whom."

You can converge on answers to those questions in a lot of ways. The fake paper detection method is suggested as one tool to aid journals tackle fraud.

If you don't think the conditions are valid, well, ok. But why not? How would you improve on the validation methodology? Obviously having more known fakes would be nice.

Saying the article is "bigoted nonsense" doesn't make a lot of sense without more information (to be fair, I might be lacking crucial context). Are the authors known bigots with history of pushing bigotry? What I read seemed to be a sincere attempt to improve scientific publication practices by identifying the scope and scale of the fraud problem, while also developing means to address it. That doesn't strike me as bigoted nonsense.

That said, the headline of the article is pretty click-baity, and shame on science's editors for that.