It does come up again and again - in this conversation, but also in mainstream and social media, as a representation of the overwhelming consensus of scientific agreement, which may plausibly affect the opinions of average citizens.
Considering this, I think it's fair to questions whether the "97% of scientists" claim is actually true, or whether it may be the result of some fancy footwork.
> I did not do this so far because it's ultimately beside the point.
Beside "a" point, but beside "the" point? Depends on the topic of discussion I suppose. In this thread, we have a prior comment:
> In fact the "95% of scientists agree" is just such an example - if you look into it, the number becomes very questionable.
So no, it isn't actually "beside the point".
> Climate science studies the climate, and not climate scientists, and I don't find the exact percentage of climate scientists agreeing or disagreeing with something at all interesting or relevant.
Me neither, but I doubt the same applies to the general public, and I know it doesn't apply to large numbers of people on social media (where I see the statistic quoted as some sort of proof, regularly), and I also know it doesn't matter to actual deniers, conspiracy theorists, and right wing media talking heads (Fox news, bloggers, YouTube celebrities, T_D members, etc) who will look for any weakness in your game and exploit it to it's maximum potential. Don't underestimate this last group - meme magic is real.
> Well this is not a popularity contest
Public support of climate change initiatives is very much a popularity contest, of sorts. The public discussion certainly doesn't consist of qualified discussion of the science.
> it's a scientific issue where there is a well-known method for making progress: You present logical arguments and counter-arguments, based on facts from credible sources (or original experiments). That's the only thing I'm interested in, and the he said / she said style of arguing that may get you points on reddit and twitter is super boring to me.
Your relative interest is interesting, but beside the point. The topic of this sub-thread is very clearly about trustworthiness of claims, so please stop complaining about the topic. If you are not interested, please don't reply.
> This is just how science works. What else are you asking for?
I wouldn't mind hearing your thoughts on the perspective below, with respect to whether it is truthful or necessary to describe the actual facts below incorrectly, as: "97% of scientists agree...", especially considering scientists numbers weren't included in the initial report.
(Note that I put a few qualifying comments below the stats part, you might want to skim those before starting a reply. Also note that we're obviously talking past each other at this point to a large degree, and I fully appreciate that this criticism is ~ridiculous to some degree (I'm not a complete idiot), but as I willingly concede that, might you willingly concede that there very much is a propaganda war going on in parallel to the actual science? Whether you personally care about that is orthogonal to the fact of its existence.)
--------------------
https://iopscience.iop.org/article/10.1088/1748-9326/8/2/024...
> We analyze the evolution of the scientific consensus on anthropogenic global warming (AGW) in the peer-reviewed scientific literature, examining 11 944 climate abstracts from 1991–2011 matching the topics 'global climate change' or 'global warming'. We find that 66.4% of abstracts expressed no position on AGW, 32.6% endorsed AGW, 0.7% rejected AGW and 0.3% were uncertain about the cause of global warming. Among abstracts expressing a position on AGW, 97.1% endorsed the consensus position that humans are causing global warming. In a second phase of this study, we invited authors to rate their own papers. Compared to abstract ratings, a smaller percentage of self-rated papers expressed no position on AGW (35.5%). Among self-rated papers expressing a position on AGW, 97.2% endorsed the consensus. For both abstract ratings and authors' self-ratings, the percentage of endorsements among papers expressing a position on AGW marginally increased over time. Our analysis indicates that the number of papers rejecting the consensus on AGW is a vanishingly small proportion of the published research.
Popular claim: 97% of scientists agree with the consensus on anthropogenic global warming.
Actual Facts:
Within the subset of climate change abstracts that mention the topics 'global climate change' or 'global warming':
66.4% of abstracts expressed no opinion on AGW
32.6% of abstracts endorsed AGW
0.7% of abstracts rejected AGW
0.3% of abstracts were uncertain
Within the subset of these abstracts that expressed an opinion on AGW, 97.2% endorsed AGW
Percentage of scientists who endorse AGW: NOT STATED
Percentage of scientists who have expressed an opinion on AGW, who endorse AGW: NOT STATED
--------------------
Please note, my question is not in any way about whether this anomaly should be considered a reasonable and valid debunking of AGW science itself (it isn't, of course), it is strictly about the truthfulness (or propaganda, depending on how one's boat floats) aspect of it.
Also note that I have no idea if this is article is based on the most up to data, or if a newer study is available that fixes the shortcomings below. If there is, fantastic, perhaps I'll read it. But again, this is not my complaint: my complaint is the "97% of scientists...." memes that were deployed into the public arena of perception. I do not like this approach for several reasons, one of which is it is susceptible to justified attack by deniers and conspiracy theorists.
You phrase it as ...
> Within the subset of these abstracts that expressed an opinion on AGW, 97.2% endorsed AGW
... which is a misleading way of framing this, because opinion does not play a role in this. Expressing an opinion in a scientific abstract would be highly frowned upon. The question is whether the abstract contains information relevant to the question of whether AGW is happening or not. These are the only abstracts that you want to consider when the question is whether the science agrees or disagrees with AGW.
Your point that going from "97% of abstracts" to "97% of scientists" is not strictly correct is fine (and this is not done in the Cook et al. paper), but this simplification in the media is not worrying to me at all, since you'd expect those to be very highly correlated.
> might you willingly concede that there very much is a propaganda war going on in parallel to the actual science?
Well, based on this comments section that seems rather self-evident. The scientists, however, are not fighting a propaganda war. That's a political or media phenomenon. Everything written in this comments section so far for me has reinforced the picture that this "war" is very one sided, since the anti-science arguments either just disappear or boil down to inconsequential pedantry as soon as you shine some light on them.
> Because absence of evidence is not evidence of absence, abstracts expressing no position can not be included in the 97% number.
This seems reasonable, but once again you're overlooking the actual claim that's being made: "97% of climate scientists...."
Now of course, from a pure scientific perspective, it's a moot point, and our entire argument is a waste of time. But this particular discussion, and the comprehensive issue of climate change itself, is not purely a scientific issue.
> You phrase it as ...
>> Within the subset of these abstracts that expressed an opinion on AGW, 97.2% endorsed AGW
> ... which is a misleading way of framing this, because opinion does not play a role in this.
At first glance, this seems rather surreal. I am stating it accurately (this wording more accurately resembles what was actually studied), but your mind perceives this more accurate statement as "a misleading way of framing this". So what's going on here?
Well, there are (at least) two differences here that I can see:
- we are using two different verbs: stating, and framing
- we are evaluating the truthiness of the different ways of stating it from (at least) two different/composite perspectives: I am looking at it from the perspectives of pedantic accuracy (how closely the description of the study matches the actual study) and with respect to vulnerability to propagandic exploitation (full disclosure: I imagine there's some motivated reasoning going on), whereas you are looking at it more from the perspective of ~"what is the most concise way these results can be stated in order to communicate the scientific facts"
So if we look at it this way, it seems to me we're both right. I don't disagree with your framing, as it relates to the science only, and I suspect (based on your prior comments) that you don't disagree with my framing, but only as it relates to pedantic accuracy of describing what the study actually did, with the intention of minimizing the attack surface to propagandists. (Do I suspect correctly?)
> but this simplification in the media is not worrying to me at all, since you'd expect those to be very highly correlated
> The scientists, however, are not fighting a propaganda war. That's a political or media phenomenon. Everything written in this comments section so far for me has reinforced the picture that this "war" is very one sided, since the anti-science arguments either just disappear or boil down to inconsequential pedantry as soon as you shine some light on them.
And here is where I wish you could try (I don't think you are) to understand what I am trying to get through to you: your focus on only the pure science aspect of the issue, and your side "winning" that war in the online culture war, is detrimental to your cause. You can have the science and technical solutions all figured out (and it is, well enough), but if you are unable to assemble adequate public consensus & support, the entire initiative may fail, not unlike a typical IT project with strong technical people but terrible analysts and project managers. If the topic of the original article was running successful IT projects, I can't imagine what I am saying right now (bad management team >> strong technical team) would receive noteworthy resistance on HN, because we've all seen it many times. But change the topic to something in the culture war category, and people's thinking changes.
>At first glance, this seems rather surreal. I am stating it accurately
You may have slightly misunderstood what I was trying to say here, I was referring only to your use of "no opinion" versus the paper's "no position." It's not a hugely important point so I'm only going to comment on the following ...
> You can have the science and technical solutions all figured out (and it is, well enough), but if you are unable to assemble adequate public consensus & support, the entire initiative may fail, [...]
If this was all you'd been saying, we'd be in full agreement. I agree there's room for improvement in popular science reporting. I disagree that there is anything majorly wrong with it, though. [0] If you have any specific recommendations for improvements that could be interesting.
However, I feel you are holding popular science journalism to an unrealistic standard here - a standard that is applied to no other subject.
If journalists were forced to use the highly technical, precise and nuanced language of peer-reviewed scientific publications in their popular reporting, that just wouldn't work for the vast majority of their readers. It would leave the field wide open to dishonest sound-bite style reporting by people unconstrained by truth. Therefore, simplifications are to some degree necessary, as long as they don't deform the underlying science. People are still free to dig deeper into the literature if they are that interested, just as we've been doing here. I don't think that's too much to ask - or, put differently, if this is too much to ask, then the problem is a much deeper one: a lack of critical thinking, a lack of curiosity, a pathological preoccupation with just the most superficial aspects of a topic.
Or, putting this a bit more practically: I feel like I have done a fine job of showing why quoting the 97% number is not particularly misleading. What form could this argument take that would fit in less than a couple of paragraphs? Because there aren't very many places outside of HN where that would fly...
[0] I don't know if this applies to places where people people mostly argue in meme format. It may not, but that would be a general cultural problem and not one of scientists and / or science journalists.
From each of the two distinct perspectives I mentioned?
> However, I feel you are holding popular science journalism to an unrealistic standard here - a standard that is applied to no other subject.
Telling the truth? Although you have a fair point, hardly anyone bothers themself with the truth nowadays.
Personally, if I was a scientist, I'd get a bit upset with the media misrepresenting things I say, especially if I believed there was chance that it could cause harm. But then, I seem to be about the only person concerned about throwing fat pitches to anti-AGW propagandists, so maybe I'm wrong.
> Or, putting this a bit more practically: I feel like I have done a fine job of showing why quoting the 97% number is not particularly misleading.
You also did a fine job at completely overlooking the two perspectives part I pointed out, that explained fairly precisely what my concern was.
Just as an experiment, I am going to copy paste that part in here to see if you will ignore it again. While this is disrespectful, I've performed this experiment many times on reddit and people will continue to pretend they don't see a point being made. Sometimes I even add over the top bolding and all sorts of other nonsense to make the spectacle as absurd as possible, to see to what lengths people will go.
But I digress.....here is the point I am curious whether you are able to address:
(HOWEVER: I already said in my prior comment for you to feel free to not respond, so I will not consider a lack of response to this comment as any kind of an internet victory. That said, I really would like to hear your response, because I'm not joking, I really do consider this perspective important, and if you truly find it (the propagandic exploitation angle) utterly unconcerning, I would very much like to know that.)
My prior point, on multiple perspectives:
-----------------------------
So what's going on here?
Well, there are (at least) two differences here that I can see:
- we are using two different verbs: stating, and framing
- we are evaluating the truthiness of the different ways of stating it from (at least) two different/composite perspectives: I am looking at it from the perspectives of pedantic accuracy (how closely the description of the study matches the actual study) and with respect to vulnerability to propagandic exploitation (full disclosure: I imagine there's some motivated reasoning going on), whereas you are looking at it more from the perspective of ~"what is the most concise way these results can be stated in order to communicate the scientific facts"
So if we look at it this way, it seems to me we're both right. I don't disagree with your framing, as it relates to the science only, and I suspect (based on your prior comments) that you don't disagree with my framing, but only as it relates to pedantic accuracy of describing what the study actually did, with the intention of minimizing the attack surface to propagandists. (Do I suspect correctly?)
I did not do this so far because it's ultimately beside the point. Climate science studies the climate, and not climate scientists, and I don't find the exact percentage of climate scientists agreeing or disagreeing with something at all interesting or relevant. Worse, harping on about that point makes it sound as if this was ultimately about some sort of popularity contest.
Well this is not a popularity contest, it's a scientific issue where there is a well-known method for making progress: You present logical arguments and counter-arguments, based on facts from credible sources (or original experiments). That's the only thing I'm interested in, and the he said / she said style of arguing that may get you points on reddit and twitter is super boring to me.
I've tried to make this point in various ways already, but it seems to be, possibly willfully, misunderstood, so I'm going to actually comment in detail on "refutations" of the Cook et al. 2013 paper [1].
The criticisms that I could find in the discussion section are that
1. paper abstracts are manually interpreted
2. there might be selection bias
3. while scientists might agree, they might disagree on the degree to which warming is occuring or to which it is a problem
Regarding 1.: Partly correct, misleading. Manual interpretation is a perfectly fine way to deal with natural language data. However, Cook et al. did not rely on this method alone. They also asked authors to self-assess. They compare their manual abstract rating to the self-assessment, and find the self-assessment to lead to substantially less "No position" ratings and more "Endorsment" ratings, while barely changing the "Reject" ratings.
Regarding 2.: Obviously correct. This is a potential issue in any empirical sociological study. In this case, however, the issue is limited, since 1) the total pool of climate scientists is limited, 2) the sample of papers (11944) and of scientists asked to self-assess (8547 asked, 1200 responses) is high and 3) the result is consistent with that of other similar studies [2].
Regarding 3.: While I agree that this would be more interesting than just distinguishing between rejection / endorsment / no position, it simply wasn't the question that Cook et al. were looking at.
Importantly, none of these points are particularly insightful, yet they were presented as if they constituted some sort of hidden knowledge uncovering a vast conspiracy. Points 1 and 2 are discussed by Cook et al. in their Discussion section, right there in the original paper. Point 3 isn't discussed because it's outside the scope of the paper.
So sure, you can now go and say "they used a sampling strategy, so there is the potential issue of sampling bias" and that is technically the truth. It is also incredibly disingenuous, since every single scientific work ever published has limitations. These limitations are understood and discussed by the scientific community. As long as no major flaw is found that invalidates the findings, the result stands.
This is just how science works. What else are you asking for? A mandatory self-assessment by every climate scientist in the world? Taking a sample and reporting statistics on the sample (while discussing any potential limitations to your sampling strategy) is the correct way to do it. It is therefore totally fine to say that 97% of scientists agree with artificial global warming, based on the study by Cook et al.
Depending on how exactly you ask the question, who you ask, and when you ask, you may get other results than 97% (Cook et al. have reviewed this in detail in [2]). That is not a contradiction, it is totally expected.
[1] https://iopscience.iop.org/article/10.1088/1748-9326/8/2/024...
[2] This is reviewed in Cook et al. 2016, https://iopscience.iop.org/article/10.1088/1748-9326/11/4/04...