That some papers are bollocks is not a controversial notion, but I genuinely have no idea if it's 10%, 50%, or 90% that's bollocks. Even just a rough indication of "about 20%" or "about 80%" would be hugely helpful. It's also unclear that non-STEM is more bollocky than STEM – Oxman's thesis is STEM, and also bollocks. STEM word salad papers don't make the news because the conclusions are rarely interesting, whereas sociology and psychology papers do, because the conclusions tend to be more interesting.
Adam - Their algorithm is inefficient. Their paper can be summarized as "use the signal-to-noise ratio," but the key idea is hidden in a page-long paragraph halfway through the paper.
Attention - I probably read "keys, query, values" a dozen times before I realized the key and query matrices multiply with the same vector (where the "self" comes in).
Latent Diffusion - Their theoretical grounding was just completely wrong. It's not some stochastic diffeq, or Markov process, or w/e terms they threw around. It's just finding the gradient of log-likelihood using finite differences (the error they add).
Of course, that applies for STEM fields as well as the social sciences. However, if we do comparisons across fields, such as looking at how many citations a typical paper in some field will receive, we find that STEM tends to do a lot better than the social sciences[1].
>Pick a field and take it down if you can. But trying to build a case against all of social science using the example of someone notoriously elevated despite a lackluster career as a publishing academic doesn't get you anywhere!
The longer such fraud goes uncorrected, the more rewarded a person is for their fraud, and indeed the more lengths other people go to cover up such fraud (as was argued by the OP) the more likely such fraud is normalized in a particular field[2]. The Claudine Gay story suggests that, if anything, XKCD was wildly optimistic about the social sciences[3].
[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5446096/ [2] https://danluu.com/wat/ [3] https://xkcd.com/451/
Sure, in the case of a very small and new field with promise of intellectually stimulating work and structural advantages for eliminating bullshit, adding more people and papers is likely going to be a net benefit. But realistically many established disciplines can only add people by lowering standards.
What I understand as the central point of your original comment was that we cannot condemn all of the social sciences (or 90% of the work in them) on the basis of the Claudine Gay story. I refute that with this sentence:
>The longer such fraud goes uncorrected, the more rewarded a person is for their fraud, and indeed the more lengths other people go to cover up such fraud (as was argued by the OP) the more likely such fraud is normalized in a particular field. The Claudine Gay story suggests that, if anything, XKCD was wildly optimistic about the social sciences.
Put another way, in the spirit of Scott Alexander's recent essay "Against Learning From Dramatic Events", the Claudine Gay story represents a significant deviation from what would be predicted by a model that assumed the social sciences had a more robust level of intellectual integrity.
the preliminary estimation phase predicts like 50% rates of replication
let's be real, the average social science paper is not good
edit: there are lots of garbage papers in every field, the culture of academia is deeply flawed, the low standards are just even lower in some fields
And by replication studies I mean "take this paper, try to replicate, write a paper on it".
Every singular paper can be wrong, that's why you're supposed to replicate them - that's at least what was taught in psychology, which many consider "social science of less rigour" - well, it was probably the course most hardcore about experimental design in my education and ability to replicate a study...
>And by replication studies I mean "take this paper, try to replicate, write a paper on it".
You don't necessarily need to perform a straight-up "replication study" in order to do some useful sanity checking on published results. It can be done as part of another novel study that builds upon prior work. That's basically what happened with the fraudulent UCLA political persuasion study several years ago [1]. Some Berkeley researchers wanted to extend the ideas in that earlier published paper, reused similar methods under the assumption that the UCLA folks were correct, and couldn't get anything reasonable within the ballpark of what was published.
[1] https://fivethirtyeight.com/features/how-two-grad-students-u...
Conservatively 90%? Bullshit. The real numbers are already terrible so why use unintentional hyperbole to make the argument it just weakens it.
Anyway, jumping from "50% of papers will probably fail an absolute bare minimum test" to "90% of papers are garbage" is subjectively reasonable to me.
If 90% seems shocking to someone, I suggest they skim a bunch of papers and develop their own feel for how bad the situation is.
Back to the main subject Vertasium has a related video:
Is Most Published Research Wrong https://youtu.be/42QuXLucH3Q?si=p9M0ZOAdHuowD30n
Even if i agreed with your "subjectively reasonable" idea, the issue is that non-STEM science encapsulate a lot of different fields.
I feel like my position is fair for any field publishing papers with falsifiable claims based on evidence?
I like the idea that he calls this a "conservative" estimate. That sounds much better than "factoid I pulled out of my butt". An unfathomably vast number of papers are published in every mainstream field of academia, STEM and not, every year. The reason we only have "conservative" "estimates" is that nobody can possibly read all of them.
Pick a field and take it down if you can. But trying to build a case against all of social science using the example of someone notoriously elevated despite a lackluster career as a publishing academic doesn't get you anywhere!