(Amusingly, they also print quotes from the same reviewer twice on that paper, creating the impression that multiple reviewers responded positively to it; I'm not going to take the time to audit all their peer quotes, but I'll bet that's a trick they applied repeatedly.)
Another example: Hypatia is apparently quite well known in feminist studies. The authors got a paper accepted in Hypatia. The authors also suggested that Hypatia was "surprisingly warm" to a paper (actually, it turns out, an essay) that suggested that white men be forced to sit on the floor in chains in classrooms. What the write-up doesn't tell you is that that's not the paper Hypatia accepted; they accepted a much more banal paper about the legitimacy of academic hoaxes.
Another example: they highlight (and so does their press coverage) a paper that spins a section of Mein Kampf into a critical race/gender theory argument. What they don't say up front is that they tried this three times, once with a relatively serious journal, once with a journal with an impact factor of 1.1, and finally with a 0.8. It's the 0.8 that accepted them; the others rejected. In fact, academic science --- STEM included --- is littered with bullshit journals, but generating that fact as the result of their "experiment" wouldn't have made for such an interesting story.
Kieran Healy makes an interesting point about this on Twitter: it's probably easier to get totally fake papers accepted than real ones. Recall Matt Might's cartoon essay about PhD theses and the tiny dent any one paper makes at the boundary of a science. If you aren't constrained by reality, it's easy to generate a paper that seems impactful. Meanwhile, the whole enterprise of peer review relies on good faith: reviewers are donating their time, and for every paper published, there are many many more that had to be read and rejected.
He first states that the particular paper's flaw is the use of implausible statistics, then goes into more detail:
"There was also considerable silliness including claiming to have tactfully inspected the genitals of slightly fewer than 10,000 dogs whilst interrogating owners as to their sexuality (“Dog Park”),"
Academia is set up to tackle people who fabricate their results - the reputational damage would destroy most people's careers. But that mechanism is not some sort of fact-checking investigation by the peer reviewers.
Let me put the counter to you: That paper has been cited precisely 0 times according to publications website. In fact, the only references I can find to it are non-academic websites which generally trawl research for funny papers they can write jokey articles about. So what has this told us about Academia?
This is an important point, and slivym shouldn't be downvoted for making it. Many people outside academia seem to have unrealistic expectations of the peer review process. Reviewers can't, in most cases, verify experimental results. Ironically, this is especially true of the so-called "hard" sciences, where a typical experiment might take months or years of preparation and cost lots of money to carry out.
The only real protection against fabricated results, in any field, is the honesty of its practitioners. What we have here is a group of rather naive people discovering that it's easy to lie and get away with it.
Disagree , and i think slivym has confused peer review with sloppy view, or perhaps peer review is indeed sloppy in their field. Yes, peer review is a bad system but it is not "nothing". Reviews won't redo the experiment but they may ask for a lot of work in reviews , they can be opinionated , disbelieving, and this is good, it keeps a certain baseline, and in my experience always improves the paper. You can easily tell if a manuscript that has been reviewed or first-submitted. It s not a matter of black and white: peer review sits somewhere in between but it's not black.
For example, my (erstwhile) field is linguistics. Suppose that I review a paper about noun incorporation in Mohawk, and the author makes various claims about which kinds of nouns can and can't incorporate into which kinds of verbs. Not being an expert in Mohawk, I can't verify those claims. If the language in question is something less studied than Mohawk, it may be that there is no reviewer available who can verify the claims without making an impractical expenditure of time and effort.
At some point, you just have to rely on the fact that most people are not fundamentally dishonest. It's like that in any field. Reviewers work for free, and aren't going to spend months verifying the results of a complex experiment.
If you didn't trust by default, you'd never publish anything.
You can still sanity-check the results, even without redoing the experiment. For example if the average agreement to a certain question on a 1-5 scale among a cohort of 10 people is reported to be 3.26, you might want to ask for the raw data, because that average is only possible with fractional answers.
I recall a study looking at such impossible aggregate statistics leading to several retractions of articles whose data had been made up outright.
Similarly, when someone claims that "four men watched 2,328 hours of hardcore pornography over the course of a year and took the same number of Implicit Association Tests", you might realize that 2328 hours/(4*365) > 1 hour 36 minutes per day; and ask for the titles and duration of the porn allegedly watched, just to make sure that this extremely onerous experiment has actually been performed.
Note that the paper about that "experiment" was not accepted , but at least one reviewer actually recommended less data ("My first piece of feedback on how to make this hybrid article work is that they should remove the quantitative data."), perhaps due to a misunderstanding of sample sizes ("It makes no sense to undertake quantitative analysis for four people – when you flatten the detail out of a sample of four you’re not left with anything interesting.") — the real sample size is at least 2328.
I realize that peer review mostly doesn't operate at that level of scrutiny, but maybe it should. Checking the raw data requires slightly more work of both reviewers and honest authors, but increases the workload of dishonest authors from "make up a few numbers" to "make up as many numbers as if'd actually done the work and don't introduce statistical anomalies", shrinking the gap to "actually do the work".
So even though you need to trust authors a little, it's certainly possible to trust less. There is no perfect protection against academic dishonesty, but there could be better protections.
I don’t see the point. The authors could easily respond with a long list of porn titles. (And as an unpaid reviewer with lots of real work to do, are you going to bother verifying every title in the long list?)
>Note that the paper about that "experiment" was not accepted
Then it's not a very good example to base your argument on.
>the real sample size is at least 2328
You’re both wrong. You can’t treat 2328 observations from 4 subjects the same way as 2328 observations from 2328 subjects (see e.g. https://en.wikiversity.org/wiki/Advanced_ANOVA/Repeated_meas...)
More generally, virtually no-one understands statistics. Every field where statistical analysis is used routinely publishes papers that use bad statistical methods.
>I realize that peer review mostly doesn't operate at that level of scrutiny, but maybe it should.
What does the “should” even mean here? Do you think that reviewers who work for free “should” do even more work than they do already? Or that journals “should” force reviewers to do this (even though they have no mechanism for doing so)? There are practical limits to the amount of scrutiny any given paper can be subject to. It would suck if we needed to spend more time reviewing papers just because a bunch of assholes keep trying to get fake papers published.
>it's certainly possible to trust less.
Not really. You don't seem to realize that more scrutiny during the review process would require real people to give up more of their real time for free. You can't just snap your fingers and make that happen.
It should be enough to randomly sample a subset for verification, similar to probabilistic proof checking in cryptography.
>>Note that the paper about that "experiment" was not accepted
> Then it's not a very good example to base your argument on.
You're welcome to suggest a better example ;)
> You’re both wrong. You can’t treat 2328 observations from 4 subjects the same way as 2328 observations from 2328 subjects
You're right of course, but it really depends on how you want to generalize. Observing only 4 subjects makes it hard to estimate population variance and generalize to other subjects, but having 2328 observations of the same subject should give great insights into measurement reliability and changes over time, for those subjects.
> Do you think that reviewers who work for free “should” do even more work than they do already?
I think that reviewers should be compensated adequately for their work, ...
> Or that journals “should” force reviewers to do this (even though they have no mechanism for doing so)?
... by the journals, which can use some of the revenue they make selling subscriptions to enforce a quality standard for the papers they publish.
> It would suck if we needed to spend more time reviewing papers just because a bunch of assholes keep trying to get fake papers published.
Some assholes try and succeed at publishing fake papers, some of them potentially influencing important decisions, e.g. in medicine. You can of course decide that it's not worth the effort to try and stop them, but I feel that publishing fake results should be as hard as possible.
> You can't just snap your fingers and make that happen.
But I can argue on the internet about it. Maybe that doesn't change anything, but it makes me feel better.
You mean, I'm welcome to find supporting evidence for your argument? Shouldn't that be your responsibility?
>I feel that publishing fake results should be as hard as possible.
Making it "as hard as possible" would mean using almost all of the world's resources to try to stop fake results being published. If you really want to make the review process more rigorous, you need to present a concrete plan specifying (a) who's going to do the work and (b) who's going to pay for it.
If you can't do that, then what makes you so sure that the review process isn't already as rigorous as it can reasonably be, given the reality of human frailty and limited resources?
Fair enough.
I had some trouble finding the original better example I had mentioned earlier ("I recall a study looking at such impossible aggregate statistics leading to several retractions of articles whose data had been made up outright."), but while trying to find it, I stumbled on another.
Example 1: https://andrewgelman.com/2018/08/21/scandal-isnt-whats-retra...
A study gets published finding a large effect, large enough to cause a replication attempt: "The effort to replicate the original study was successful in everything except the creation of the PSU-level structural stigma variable."
The suspected reason for this replication failure (imputation of missing data) turns out to be wrong when the original authors have someone check their code for data analysis, which is found to contain an error.
If that code had been checked during peer review (or at least afterwards, by including it with the publication), the effort would have been less than a full-blown replication attempt.
Example 2: https://medium.com/@jamesheathers/the-grim-test-a-method-for...
(Which is what I was referring to earlier.)
Simply checking reported means and sample sizes for consistency revealed mathematically impossible results in 50% of tested papers.
He goes to great lengths to stress that such inconsistencies don't necessarily imply fraud (some are honest mistakes), but the behavior of some of the contacted authors when asked for their data appears very sketchy.
Again, if there were a culture of looking at raw data and inspecting analysis code during peer review, those studies reporting obviously incorrect results would not have been published, saving everyone who relies on such studies a lot of trouble.
So, who's going to do that work and who pays for it? I'd be surprised if I managed a working plan on the first attempt, but here's my proposal:
- Authors prepare their data and code together with instructions in such a way that an expert in their field can work with them without having to ask the authors for additional information. It should be attached to the paper as supplementary material. If the data is privacy-sensitive, it should at least be made available to reviewers to check that the results follow from the data. Who pays for it: whoever pays the authors to be writing papers in the first place.
- Reviewers do that sanity-check of running the code on the data to verify that the instructions are complete and the results match what is reported in the paper. They scrutinize the code to the level they'd apply to a methodology section. Who pays for it: the readers of the published paper, since they benefit from not having to do the peer review themselves when they just want to use the results.
Maybe that's unrealistic and the review process is
> already as rigorous as it can reasonably be, given the reality of human frailty and limited resources
but that would be sad.
I can't make sense of this. Are you suggesting that journals should pay reviewers and finance this by charging people (more) to read articles?
As for reviewers checking statistical analyses, remember that this is peer review. The reviewers, on average, are going to be just as sloppy and ignorant and careless as the authors. If a field is awash with papers with bad stats, then most reviewers (being drawn from same same pool of people) will not be competent to check a paper's stats. Andrew Gelman isn't available to review every social science paper.
Further: would you rather have peer reviewers be established experts in their fields, or new grad students? If the former, how does reconstituting peer review to trade expectations of good faith for diligent, expensive, tedious review impact who will end up willing to do peer review?
Because of that incoherence, it's not what I want. I support paying reviewers for the important work they do. If the research is of a kind that should be available to the public for free, then that's a job for research-funding organizations, who could explicitly pay for that cost (rather than implicitly by granting funds to scientists who then end up working as reviewers for free).
Indeed there is nothing wrong with any concept. If they wanted to show some systemic bias they should also publish a study about how to train women like dogs to attack men.
> Reject studies that have implausible findings?
Yes (if by implausible you mean unconvincing). Reject and resubmit with more powerful statistics.
> or publish it and let people follow up and do their own studies
Indeed , like good software, you publish a barely functioning version and let others find the bugs /s
> authors are welcome to suggest anything that like as a result of their research
Not really, editors will often tone you down. Otherwise we might do away with peer and editorial review.
I think you are advocating for 'anything goes' publishing here. Sure but then these journals should not call themselves scientific, it's a travesty to co-opt that name to validate their existence. Which in the end proves the point the author makes.
The author clearly has an agenda here, that's why they re publishing a blog, not a "science journal"
I should clarify here- I mean implausible by the standard of common sense, but backed up by the statistics presented in the paper, which was the case here.
Given how long this article is, it doesn't seem likely they were omitted for length. They're trying to present a one-sided slant of the peer reviewers as overly credulous.
1. That they were published with obviously impossible results (inspecting the genitals of 10,000 dogs while asking owners about their sexuality).
2. It's where they were published. e.g. In the #1 "best" feminist journal Hypatia.
3. It's the evil in the messages they got published. They got a 3000-word except of Mein Kampf published, by rewording it in the language of intersectional theory. They got published recommending that "privileged" students be physically chained and silenced during class. They got published by proposing men be trained like animals. (This is not the same as learning about human behavior by studying dogs; it is to suggest that it's okay to treat people like dogs if they're born a certain way).
All this demonstrates a field of study that is not just soft in terms of intellectual rigor ("implausible"), but is willing to casually support reprehensible policy ideas if they're framed within a specific neo-Marxist belief system and targeted at men or white people.
Bear in mind that many of the people producing the papers in these fields receive public funds at public institutions. They project an image of being neutral scholars to justify those funds. What this hoax has done is given evidence that they are not neutral scholars. They are moralizers, activists, revolutionaries working to target a specific group of people, and are not very concerned about the rigor of the intellectual pathways they use to get there.
A group of people dedicated to a process are called scientists. A group of people dedicated to a conclusion is called a religion or an ideology. Social scientists claim to be the former. But the hoax papers demonstrated that, at least with some regularity and even in the highest journals, the social sciences act like the latter.
That submission actually was not published.
Why is that 'obviously impossible'? How long does it take to look at a dog's genitals, 5 minutes?
I haven't read the paper, but you could get a team of 10 undergrad (or grad) students, and send them out to different dog parks in the area. If each student inspects 20 dogs in a day (say they go to the park for a few hours--I assume many owners will say no) then you've got your data after 50 days of work.
Say your students are only free on weekends (when dog parks are likely more busy, anyway) then it'll take you a little more than six months to gather all your doggie-dick-data.
That's not at all an unreasonable length to run a study.
Elsewhere: "From 10 June 2016, to 10 June 2017, I stationed myself on benches that were in central observational locations at three dog parks in Southeast Portland, Oregon. Observation sessions varied widely according to the day of the week and time of day. These, however, lasted a minimum of two and no more than 7 h and concluded by 7:30 pm (due to visibility)." It'd be an average of nearly 3 hours per day, every day.
10,000 dogs / 1000 hours = 6 minutes per dog.
Ok, so it's somewhat implausible for one person to look at this many dog-genitals, but it's certainly physically possible. How many animals does a veterinary see in a year?
I do hope you see that such a thing would not be a reasonable demand on these people. That's WAY too much. That's literally 1/5th of their entire school year.
> Thesis: That academic hoaxes or other forms of satirical or ironic critique of social justice scholarship are unethical, characterized by ignorance and rooted in a desire to preserve privilege.
This is obviously something that can be argued, even taking this very same project as an example! But for the authors to claim that the position is absurd begs the question in favor of their project. That looks disingenuous to me.
Reading through the methodology I'm really struggling to see what the actual value of what they're doing is. For example - the Dog Park study:
>What if we write a paper saying we should train men like we do dogs—to prevent rape culture? Hence came the “Dog Park” paper.
If you want to assert that this is a nutty idea - please back up that claim. Because I feel fairly comfortable arguing that we can learn a lot about social constructs by studying how they differ in other species. There's nothing wrong with that concept. Maybe there's something glaringly wrong in that particular paper, but these authors do nothing to actually demonstrate that.
Later on that go into detail with this. That paper is meant to have implausible statistics. Ok, what is peer review meant to do? Reject studies that have implausible findings? We have a word for that - Bias. The correct response to a paper with implausible findings is to publish it and let people follow up and do their own studies. That is the system working correctly.
As for advocating for outlandish actions as a result of the study. Well authors are welcome to suggest anything that like as a result of their research. Publishing it doesn't imply endorsement. It implies "This is a real study".
So let's talk science. I have a confounding variable - bad studies got published because it's better to publish lots and let the academic community decide the merits of the study than to censor what you think is either implausible or has troubling parts. That explains all the results of this study without the wildly insulting and vaguely conspiracy theorist assertion that these papers get published because of an agenda on the part of academics.