News for hackers. Stuff that matters.
Re:What's wrong with social science and how to fix it (2020) by matthewdgreen on Sunday December 11, 2022 @03:12PM (#33946968)
Reading through this quickly I see several problems that make me wonder if the author has worked in academia.

1. Publication venues aren’t created equal. People outside of academia don’t understand that anyone can and will start a journal/conference. If I want to launch the Proceedings of the of Confabulatory Results in Computer Science Conference, all I need is a cheap hosting account and I’m ready to go. In countries like the US this is explicitly protected speech, and the big for-profit sites run by Elsevier et al. will often go ahead and add my publications to their database as long as I make enough of an effort to make things look legitimate.

In any given field there are probably a handful of “top” venues that everyone in the field knows, and a vast long tail ranging from mid-tier to absolute fantasist for-profit crap. If you focus on the known conferences you might see bad results, but if you focus on the long tail you’re doing the equivalent of searching the 50th+ page of Google results for some term: that is, you’re deliberately selecting for SEO crap. And given the relatively high cost of peer-reviewing the good vs not peer-reviewing the bad, unfiltered searches will always be dominated by crap (surely this is a named “law”?) Within TFA I cannot tell if the author is filtering properly for decent venues (as any self-respecting expert would do) or if they’re just complaining about spam. Some mention is made of a DARPA project, so I’m hopeful it’s not too bad. However even small filtering errors will instantly make your results meaningless.

2. Citations aren’t intended as a goddamn endorsement (or metric). In science we cite papers liberally. We do not do this because we want people to get a cookie. We don’t do it because we’re endorsing every result. We do it because we’ve learned (or been told) about a possibly relevant result and we want the reader to know about it too. When it comes to citations, more is usually better. Just as it is much better to let many innocent people go free rather than imprison one guilty one, it is vastly better to cite a hundred mediocre or incorrect papers than to miss one important citation. Readers should not see a citation and infer correctness or importance unless the author specifically states this in the text at which point, sure, that’s an error. But most citation-counting metrics don’t actually read the citing papers, they just do text matching. Since most bulk citations are just reading lists, this also filters for citations that don’t mean much about the quality of a work.

The idea of using citation counts as a metric for research quality is a bad one that administrators (and the type of researchers who solve administrative problems) came up with. It is one of those “worst ideas except for any other” solutions: probably better than throwing darts at a board and certainly more scalable than reading every paper for quality. But the idea is artificial at best, and complaints like “why are people citing incorrect results” fundamentally ask citations to be something they're not.

Overall there are many legitimate complaints about academia and replicability to be had out there. But salting them with potential nonsense does nobody any favors, and just makes the process of fixing these issues much less likely to succeed.

Re:What's wrong with social science and how to fix it (2020) by darawk on Sunday December 11, 2022 @03:48PM (#33947324)
1. He addresses this repeatedly throughout the piece. Journal impact factor is (largely) uncorrelated to replication probability.

2. Yes, but this hardly seems like a defense of citing something false (without comment), or something that has literally been retracted years ago, which is a large part of his complaint.

He is not suggesting the use of citation count as a metric for quality. I have no idea how you could have possibly gotten that from reading this article. A bullet point in his "what to do" section is literally "ignore citation counts".

Re:What's wrong with social science and how to fix it (2020) by matthewdgreen on Sunday December 11, 2022 @04:13PM (#33947582)
> He is not suggesting the use of citation count as a metric for quality. I have no idea how you could have possibly gotten that from reading this article.

TFA is extremely clear that the presence of citations (in the aggregate, as a count) on “weak” papers is something the author considers a problem and a perhaps a moral failure on the part of citing authors. The author also believes that citations should be “allocated” to true claims.

* “Yes, you're reading that right: studies that replicate are cited at the same rate as studies that do not. Publishing your own weak papers is one thing, but citing other people's weak papers?” Here citations are clearly treated as a bulk metric, and “weak” is a quality metric.

* “ As in all affairs of man, it once again comes down to Hanlon's Razor. Either: Malice: [the citing authors] know which results are likely false but cite them anyway. or, Stupidity: they can't tell which papers will replicate even though it's quite easy.” Aside from being gross and insulting — here the author claims that the decision to cite a result can have only two explanations, malice and stupidity. Not, for example, the much more straightforward explanation that I mention above (and that the author even admits is likely.)

* “ Whatever the explanation might be, the fact is that the academic system does not allocate citations to true claims.” The use of “allocate citations” clearly recognizes that citation counts are treated as a metric, and indicate that that the author wishes this allocation to be done differently.

* “This is bad not only for the direct effect of basing further research on false results, but also because it distorts the incentives scientists face. If nobody cited weak studies, we wouldn't have so many of them.” Here the author makes it clear that they see citation count (correctly) as a metric that encourages researchers, and believes the optimal solution is to remove all (but perhaps explicitly negative?) citations to those papers.

> A bullet point in his "what to do" section is literally "ignore citation counts".

Yes, after extensively complaining about the fact that citations aren’t used by authors in a manner that reflects the way they’re used as a metric, then complaining further about the fact that authors do not use them this way and repeatedly urging them to change the way citations are used — the author then admits that their use of a metric is problematic and should be ended.

We agree! The only problem here was that the author took a detour to a totally absurd place to get there.

Re:What's wrong with social science and how to fix it (2020) by darawk on Sunday December 11, 2022 @07:13PM (#33949146)
> TFA is extremely clear that the presence of citations (in the aggregate, as a count) on “weak” papers is something the author considers a problem and a perhaps a moral failure on the part of citing authors. The author also believes that citations should be “allocated” to true claims.

As I see it, there are two independent properties that the author is saying ought to be dependent. And I think you (and I) actually think the same. If citations are going to be treated as a metric, then the way they are written (without regard for quality or accuracy) is bad. If citations are not going to be written without regard for quality and accuracy, then they shouldn't be used as a metric. Either one of these models would be fine. What is not fine is the present reality: Citations are written without regard for quality and accuracy, and then still used as a metric ubiquitously! Impact factors, the most common method of ranking journals, are literally measures of citations.

> Yes, after extensively complaining about the fact that citations aren’t used by authors in a manner that reflects the way they’re used as a metric, then complaining further about the fact that authors do not use them this way and repeatedly urging them to change the way citations are used — the author then admits that their use of a metric is problematic and should be ended.

The crux of your point though seems to be that nobody uses them as a metric, and I'm just going to have to fundamentally disagree with that. It's true that authors, when writing papers, appear not to give them the care that a metric would deserve. What is not true is that citations aren't used as prima facie evidence of quality/importance throughout academia.

Re:What's wrong with social science and how to fix it (2020) by matthewdgreen on Sunday December 11, 2022 @08:54PM (#33949950)
>The crux of your point though seems to be that nobody uses them as a metric

I didn't say that at all: what I said is right there in my post.

What I did say is that citation counts are a bad and noisy metric, one that is a side-effect of measuring a tool that has a very different purpose, and which researchers can use (usefully) for a variety of reasons that don't require them to validate the technical correctness of all cited works. Nor would it even make sense for citations to be used that way.

The author's criticism in TFA (which he delivers in the strongest and most explicitly moralistic terms) is that based on using this citation count metric, which he selected, the field is broken because bad work gets cited. That's his criticism and his choice of measures to make it. But since it is quite normal for people to cite work that they haven't carefully reviewed for technical correctness, this criticism is essentially bunk.

At a deeper level, the criticism fails to appreciate how people use citations as a measure for academic promotion. In most cases tenure committees care about aggregate statistics like total citations, h-index or i10-index. If a researcher publishes a work that receives hundreds citations for ten years and then fails to replicate, then it basically doesn't matter if the work stops receiving future citations. A retraction might matter. Reports of the failed replication might matter. But nobody is going to lose out on a promotion specifically because some random paper receives 8,000 citations in the first ten years and then zero citations after the failed replication.

Re:What's wrong with social science and how to fix it (2020) by darawk on Tuesday December 13, 2022 @12:05PM (#33971767)
> At a deeper level, the criticism fails to appreciate how people use citations as a measure for academic promotion. In most cases tenure committees care about aggregate statistics like total citations, h-index or i10-index. If a researcher publishes a work that receives hundreds citations for ten years and then fails to replicate, then it basically doesn't matter if the work stops receiving future citations. A retraction might matter. Reports of the failed replication might matter. But nobody is going to lose out on a promotion specifically because some random paper receives 8,000 citations in the first ten years and then zero citations after the failed replication.

This is his point, though. Not only are authors still getting tenure after failed replication, they're still getting citations! Citations that don't even mention the failure to replicate!

The fact that citations are used as a metric to get tenure is the problem. There are two solutions to that problem: Change the culture around citing things, or change the metrics people use. That is the whole point of the post.

Re:What's wrong with social science and how to fix it (2020) by matthewdgreen on Tuesday December 13, 2022 @01:43PM (#33973719)
> This is his point, though. Not only are authors still getting tenure after failed replication, they're still getting citations!

And his point is irrelevant. For two reasons that I've already explained multiple times now!

(1) As I've tried to point out (I've written three posts now to explain this, why are we still debating this!), there are plenty of valid reasons why non-replicating works might get cited. To prevent these works from being cited, you would need to fundamentally change citation practices in a manner that would harm researchers' ability to use citations for their intended purpose.

(2) As I also explained above: even if you somehow managed to force all other researchers to alter their citation practices, it almost certainly wouldn't matter for the purposes of promotion decisions anyway. For the purposes of promotion, the influence of each incremental citation drops off exponentially. After a few years of a work receiving citations, later incremental citations have at most a negligible influence on a researcher's record.

Unless replication failures happen extremely quickly, it doesn't really matter whether future researchers do or do not cite the failed work. The early citations will still exist and will vastly dominate later ones in promotion decisions. The only situation where citation-bans would help is one where citing authors could somehow intuit that a work would fail to replicate, prior to seeing an actual failed replication. (TFA claims this is easy. I think TFA is not credible.)

TL;DR: forcing the entire field to change the way they use citations is (1) harmful to researchers, and (2) despite that is unlikely to have any major benefit anyway, since later citations are not heavily weighted when promoting researchers, and replication attempts generally occur later in a work's citation lifetime.

Re:What's wrong with social science and how to fix it (2020) by einpoklum on Sunday December 11, 2022 @06:17PM (#33948711)
> Citations aren’t intended as a goddamn endorsement (or metric).

This. I may well write the equivalent of:

"So, those clowns of [18] thought they could sneak it past us to only model the near-trivial case which accounts for less than 5% of real-life occurrences (as per the distribution estimated in [5])."

The second citation is barely-endorsing, and the first citation is an anti-endorsement. Or I could write:

"A bunch of useless attempts have recently been published to achieve what I'm actually going to get right. [4, 7, 8]."

Now, I wouldn't use this language, I'd be more polite and much more circumspect, but still, those are legit and not-entirely-uncommon citations.

Re:What's wrong with social science and how to fix it (2020) by jcampbell1 on Sunday December 11, 2022 @03:58PM (#33947418)
You are only making the author seem more correct. You have a system where citations act as cookies and endorse (administrators fault) but that is not the researchers intent.

> Whatever the explanation might be, the fact is that the academic system does not allocate citations to true claims. This is bad not only for the direct effect of basing further research on false results, but also because it distorts the incentives scientists face. If nobody cited weak studies, we wouldn't have so many of them. Rewarding impact without regard for the truth inevitably leads to disaster.

You also argue that high quality journals are good at filtering quality. The author presents evidence that this is specious.

You furthermore question whether the author worked in academia, which is answered.

Given your reading comprehension skills, I wonder if you are in the correct job.

Re:What's wrong with social science and how to fix it (2020) by wizofaus on Sunday December 11, 2022 @11:07PM (#33950748)
> Just as it is much better to let many innocent people go free rather than imprison one guilty one

Why is that a "rather"? I'm guessing you mixed up "innocent" and "guilty" there, but if so, I'm not sure I could confidently agree with a claim that general (certainly not if the "many guilty" were serial ķillers that went on to commit many more homicides, vs the "one innocent" being given a community service sentence for a minor misdemeanor they didn't actually commit).

Re:What's wrong with social science and how to fix it (2020) by UncleMeat on Sunday December 11, 2022 @07:14PM (#33949156)
In fact, we even cite papers when we are saying that the work is wrong or deficient.
Re:What's wrong with social science and how to fix it (2020) by kazinator on Sunday December 11, 2022 @04:04PM (#33947485)
> Citations aren’t intended as a goddamn endorsement (or metric).

Works for GNU Parallel. :)

Re:What's wrong with social science and how to fix it (2020) by WalterBright on Sunday December 11, 2022 @05:17PM (#33948193)
> for-profit crap

And of course we all know that making something for profit means it's bad.

Re:What's wrong with social science and how to fix it (2020) by matthewdgreen on Sunday December 11, 2022 @09:04PM (#33950010)
For-profit in this case means "I pay you $2500 and you let me give you a list of the peer-reviewers I'd like to review my paper, all of whom happen to be my friends and business associates."