To roll back to your parent: "let me say that I'm not a climate skeptic ... yet"
I suspect it's unlikely, or that you would be convinced, but I make just one recommendation. If you haven't read or watched the movie of "Merchants of Doubt", you should as it's both a very revealing and very depressing read. The book has more depth, but the movie is more insightful as you get body language. Via interviews with the notable criminals, and TV clips. Like the fellow with a bloody big grin telling that he couldn't touch Hansen on the science, so took to discrediting him personally and had a good laugh about it. Or Heartland or CATO Institute's, I forget which, entirely scam inverse IPCC report - identical match on look and feel even to number of pages. Except says the opposite. With nary a science degree between the lot of them. Send it to every US representative, claim it's truth.
It's tobacco 2.0, and not limited to climate. Every method, every technique, every claim. From the same PR agencies and bullshit institutes, the same people who claimed there's no link between smoking and health now on TV claiming there's no warming. Then again that there is warming and it's a great thing, we should have more of it! etc. You might even understand why, on balance, I am more leery of industry than academia.
Check it out. Thankfully Europe is not nearly so scary yet. Even in the US leaning UK.
So, specifics:
Moving is probably the lesser cases for data adjustment of what springs to mind. There'll be transcription errors, calibration errors affecting a particular station or sensor, methodology and more general systemic errors, such as sea temperature readings as we discover the ship might have an effect on some methods, or that US Navy logs truncated some historic readings, IIRC rounding down to whole degrees only. I tried to find a link to the Navy story, but that seems thoroughly buried by denial results.
I come back to the very tedious simple question. What is in it for the Met Office to fraudulently and inaccurately rig their dataset? Their entire reputation, and the majority of income, depends on accurate weather and climate forecasts. What matters for their future is that they match with reality, otherwise they lose contracts to those who can match reality. They've made a few famous mistakes over their decades of forecasting too. So no, not perfect.
Then the multiple leaked internal oil company reports, not for public consumption. They have been accurately, surprisingly accurately, predicting effects since the 1950s. Even they have overshot some of the predictions - yet whilst they are claiming rising temperatures, I don't hear claims Exxon and BP are in the pocket of the great global climate conspiracy.
> One of the pillars of science is providing your raw data
Absolutely. Providing raw data set so others can audit and replicate, or find faults. Not presenting as the definitive raw graph to show a gullible public it's not happening at all.
Some things don't need adjusting, others require everything be. Weather recording will be somewhere in between. My only work exposure to sensors via software was liquid pressure sensors many decades ago, and entirely unrelated to climate. Every single reading bar none needed adjusting to fit the unique calibration of that particular sensor. Re-calibration was needed regularly. It was a pain in the ass, as that was a manual "send out an engineer" recalibration.
The blog overlays raw results to show there is no climate effect. Are you seriously claiming they have adequately invalidated every adjustment? Why isn't there a paper to read about in Nature? Presuming they haven't invalidated every last one, that's misleading in the extreme.
I'm perfectly comfortable that someone may find reason to suspect or exclude some data, and some adjustments. In fact I would expect it in every sizeable dataset. All? That better have some compelling basis. There's none. Just claims of fabrication.
So reading from your start point: There looks like reason for suspicion of one localised set of results. Except immediately after he's talking of disproving the hockey stick. Time to stop reading. No point digging for context, or looking to original source.
This is astonishingly well worn ground, and Muller, the instigator of that particular conspiracy theory, famously 100% recanted and has since set up https://en.wikipedia.org/wiki/Berkeley_Earth
https://en.wikipedia.org/wiki/Hockey_stick_controversy
I'm out.
I did not present the raw and adjusted data to show "there is no climate effect". They were presented to show that the observed climate effect in the 20th century is the result of adjustments to the raw data. This isn't the same argument at all: it merely sets up the next questions of what are these adjustments, why do they have such big effect and are we really sure they're correct, and why does NOAA report simulations as if they were actual measurements? Questions which are then investigated in turn. The answer may turn out to be "the code is correct, the algorithms are justified, the adjusted/synthetic data is more reliable", in which case, no problem. But given the low quality of the process these are questions absolutely worth asking.
Also these analyses aren't the old ones you're referring to from the 90s about medieval times, or even "Climategate" (which wasn't about any specific scientific claims anyway, but the process by which they were being made). This particular strand only focuses on the 20th century, and are recent analyses triggered by the rapid jump in synthetic data points being reported as temperature readings.
So reading from your start point: There looks like reason for suspicion of one localised set of results. Except immediately after he's talking of disproving the hockey stick. Time to stop reading. No point digging for context, or looking to original source.
Again I think you subtly misunderstood the argument.
The argument made wasn't "maybe there's something wrong with the data in Minnesota". That state was just used as an example because it has a good balance of AM/PM stations in the years when the adjustment is the biggest. The argument was that a key adjustment made to the data doesn't appear to be justified in for that state because the difference has an alternative explanation, yet despite decades of adjustments this was never detected? And by extension, maybe all cases where TOB adjustment is applied have these sorts of problems. To know whether TOB adjustment is fundamentally flawed would require more research, and the other adjustments even more, but there is smoke.
Your comparison to liquid pressure sensors doesn't help because in those cases you could measure the ground truth and adjust the sensor to match a pre-calibrated sensor. There's no way to know the ground truth for what temperature really was in Minnesota in the 1930s - all we have are the recorded measurements from those days, by people who thought they already had calibrated sensors. That's why we have to be very careful when playing with the data: we have nothing to calibrate against. The results of any adjustment are uncheckable except via reference to internal consistency and other explanations.
Now your position seems to be that there's really no way to investigate these models climatologists use that you will ever accept, because if the outcome is anything other than "yep, everything is basically correct +- minor changes" then you'd write it off automatically.
I understand that position: life is short, after all. But you shouldn't trash or write off people who are still investigating. As makomk points out elsewhere in this thread, a peer reviewer paper got published in Nature that promoted the current consensus view, but it turned out to be misusing data and maths to make the conclusions sound way more certain than they were. Found by a blogger who has been critical of climate science for a long time. The system doesn't always self correct and outsiders have different incentives to government-funded scientists.
As for the Met Office, of course they aren't directly incentivised to give wrong results. But we know they did before because for years temperature datasets showed a "pause" in warming nobody could explain. That data has now been erased and the latest temperature datasets show no pause ever happened. Where were the mass investigations into that? Look, if the Met Offfice reported wildly different results to the "consensus" then they'd be picking a fight with the rest of the climate establishment, which would be a huge deal for them and put their entire reputation at risk, which is very important to them. In the past I've witnessed the effects of someone claiming there's a consensus of experts supporting their view, in a case when I was deeply involved and knew flat out there wasn't. The effects were terrifying; hundreds of people slowly lost their minds and came to believe things that were clearly nonsense. There actually was no consensus about such ridiculous ideas, just a few people claiming repeatedly there was and a whole lot of social pressure / selective editing on community news channels to make it look like more people agreed with them than really did. It's very easy to see signs of the same things happening here. That's why I want more outsiders and outside scrutiny.
You yourself come across as someone with a genuine curiosity, which is why I've persisted. Yet seem sceptical mainly of government, but not so much of an entirely anonymous and unaccountable blog post. I find that surprising. Particularly when so many of the USA's "independent" voices and are shown to be industry backed.
> The system doesn't always self correct and outsiders have different incentives to government-funded scientists.
That's not in dispute, and applies to all fields. That does not imply fraud. Fraud requires additional evidence than finding an error or ten. That does not imply government funded is the least ethical or least accurate of all. How long were the major security flaws in OpenSSH? How often do we read of a zero day that turns out to have been there for ages, merely unnoticed?
Unqualified political lobbyists have different incentives too - the think tanks, PR agencies and free market institutes who shouldn't be permitted to express opinion, or spin fake alternatives, on active scientific research at all. If you want an outsider who has a vested interest to pollute our quest for knowledge, for the sake of an election or quick buck, they are it. Quite often they are committing deliberate intentional fraud.
https://www.theguardian.com/environment/2019/oct/10/vested-i...
I'm going to once again recommend a read or watch of Merchants of Doubt, even if you remain totally unconvinced by then end of it.
Incentives generally apply more in industry where profit encourages the suppression of an inconvenient study, or to engage in fraud to pass emissions tests. The history here is also very well worn, in many industries that adopted the tobacco model, so there's no point replaying it.
I step back here. We both claim to want the same thing: More accurate science and better identification and correction of flaws. There will always be flaws and retractions - that's part of the process. Neither can we introduce rigour into hundred year old measurements if we find some issue much later. Nor can we guarantee that any endeavour involving humans remains free of intentional fraud.
Focusing on government science as constant bugbear weakens your message markedly as in my experience they also have fewer incentives to falsify than industry and politics. Particularly when it is in government interest to downplay the climate crisis as they have done for the last 40 years. Institutes and lobbyists have no incentive or even legal requirement to be truthful. Doesn't mean I find government, any government, entirely trustworthy. Even so the things they fund can be surprisingly successful, and can aim for best practice. I am sceptical of all sources, I am dismissive of political sources, and most sceptical of industry sources. You are never likely to persuade a European to adopt an ingrained US hatred and distrust of government funding in and of itself, that exceeds all other funding sources. All I can say is thankfully our continents must have had very different experiences. Yet I remain sceptical of all sources.
The old HadCrut datasets 1-4 are still available for download. Changes and corrections are documented and explained.
https://www.metoffice.gov.uk/hadobs/crutem3/jan_2010_update....
What are that random blog's incentives? Entirely opaque - merely an about page that could be entirely fiction. No proof. No accountability. No corroboration anywhere. I see poor practice in presentation, poor practice in analysis, and I see attempt to take down pretty much the entire field from a skim through the pages - that's not healthy scepticism and correcting error, or advancing knowledge, that's a campaign. An agenda. Even if they do find genuine fault in some areas. From the example given there appears to be a real case to investigate on that particular region. Yet the presentations in the entirety lead me to expect to find cherry picking or sleight of hand, and I just haven't spotted it yet. Wouldn't surprise me in the least to discover "the author" is actually a dozen people inside CATO or Exxon.
I know how _I_ would make a case again some of the results if I felt I had found errors, and it would focus on the specific rather than keep widening to the general at every opportunity. If I were to conclude it widened to generality I would have presented compelling and extraordinary evidence covering the whole generality. I would not just presume it so and yell "fraud". I would encapsulate it to correlate with each set of historic corrections and attempt to show their validity or not. That would also aim to include no judgemental language. I would present it in the best way I could to encourage NOAA or whoever to respond, or investigate my finding to permit correction. Perhaps to discover a year or two later that my investigation prompted some improvement.
Even the takedown of Dr Andrew Wakefield, with his bullshit and actually fraudulent MMR study didn't have others immediately yelling "fraud" on an anonymous unaccountable blog. Others sought to replicate and see if the study stood scrutiny. Still others sought to find fault in his methods. Talk of fraud only came much later as preparation and case was made for him to be struck off. Interesting he is still free to promote and pollute the US with his lies and false medical claims, and earn high on the back of them.
Throwing in emotive words like "fake", "fraudulent" and all the others that imply deliberate intent with no proof of fraud whatsoever, are trying to sway the reader into political opposition. To take their "side". There is no "side". Present the errors, explain the errors, try and get them corrected and better practice in future. We're shooting for accuracy and improving the corpus of knowledge, not standing for election are we not?
Ah, there's the issue. I am now, on balance, convinced that site is not trying to correct careless mistakes, improve accuracy or advance knowledge. It's emotively trying to persuade - truth being secondary, or at best incidental. Even then, they may have found actual issues with some of the readings.
I really can say no more.
I used to have near absolute faith in academic scientists, with much the same reasoning as you: the government pays them merely to do research and doesn't demand any specific outcomes, so they must be less biased than anyone working in industry where there is at least pressure to produce something pleasing for their employer.
Over the years I've read a lot of research papers. Mostly in computer science but sometimes others too. In the past few years I noticed that I tended to be more enthusiastic about reading papers that came out of corporate research labs than pure academic productions. On reflection about why I felt that way, I realised it's because when I thought about the research I'd found most exciting over the years it was usually the research that solved some major (in my eyes) problem in a practical and useful way, and that such papers typically had some corporate email addresses at the top, or were at least collaborations with universities. On the other hand papers that came out of only academia often felt rather useless, trivial or like they were going down blind alleys.
Worse, a very small number of university papers unfortunately were in my eyes outright misleading or appeared to be deliberately deceptive, e.g. making eye-catching claims in the abstract that were undermined or voided by statements right at the end of the paper, or which didn't mention critical details that broke their conclusions. I'm thinking of papers dealing with security technologies I happen to be specialised in. Despite what many would have me believe, my own experience has been that corporate funded research never tried to mislead me in this sort of way, but it's happened multiple times with government funded research.
During the time these feelings were developing the whole replication crisis started. At first it was just a few psychology papers that seemed in trouble. Amy Cuddy was outed as building a speaking career on top of her flawed psychological research. I noticed that despite one of her co-researchers admitting the paper was nonsense in writing and effectively retracting it, Cuddy continued to build up her career on the back of it, aided by journalists who wrote breathless articles about this "inspiring woman", apparently not bothering to spend two minutes Googling her research to check if it replicated. Then more and more papers started getting in trouble. Whole fields of research started to disappear. Nutrition turned out to be riven with fake findings. I learned that venture capitalists avoid investing in biotech firms because of an open secret: that around half of all published microbiology papers coming out of academia didn't replicate. Earlier this year I read https://slatestarcodex.com/2019/05/07/5-httlpr-a-pointed-rev... which showed how psychiatric researchers investigated the gene 5-HTTLP for decades, published over 400 papers on their "discoveries", and then they all collapsed at once because a single paper showed that the gene couldn't possibly have the claimed effects. Literally that entire area of research was the result of misuse of statistics.
John P Ioannadis published a meta-study that appeared to show "why most published research findings are false":
https://journals.plos.org/plosmedicine/article?id=10.1371/jo...
This paper is not the result of some kind of weird cranko blogger. As far as I know, the paper had huge impact and kicked off the current replication movement in a few subfields (by no means all of them).
I did some of my own research into an economics paper that was being used to make suspicious claims on a topic I was an expert in. It had got picked up by major global newspapers, but on reading the paper I discovered grotesque amounts of scientific fraud. Then I discovered that the Chief Economist of the World Bank had been sounding the alarm about research malpractice and bogus papers in economics for years: his reward for blowing the whistle was to be fired.
Finally, there was this paper: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4816570/
It basically destroys every area of science based on doing small-scale experiments (where "small scales" means fewer than thousands of participants, i.e. nearly all of them). At a stroke that one paper wipes out huge areas of what we thought was "knowledge".
Now we see climatologists editing their data sets in such a way that the results always fit their theories. I was taught at school that adjusting theories to fit the data is science 101, that you handle uncertainty with error bars and statistics. But that entire principle has been thrown out with barely a sound. How did all this happen, and what else isn't obvious about how science happens in the real world?
The problem is that government funded academics are not free of incentives or bias, as I had once assumed. Yes, the government doesn't tell them what to find. But academics are just as strongly incentivised as any corporate-funded researcher, namely, they're incentivised to find things that are interesting. That is so vague it doesn't sound problematic, but it is. Researchers who fail to find things that are interesting don't get published and thus "perish", to use the well worn aphorism about academic success.
What defines interesting, exactly? Anything eyecatching or novel counts. Replications mostly don't. This is how psychology and biotech got into such a mess.
What defines eyecatching or novel? Anything your peers like. Who are your peers? More academics, with the same incentives. Please them and you'll be rewarded with grants, jobs and funding. Displease them and unless you win the argument so comprehensively there's no doubt possible at all, there will be no reward. You can't do what businessmen do and prove you're right via market success.
The least pleasing thing you can publish is evidence that nobody understands the field of study you're in. If the results of your research is "an economy is too complex to model at this time", or "the climate does not behave like our theories say but there are no other theories" then this is a dead end for an academic career.
The entire system is circular. I find corporate funded research into computer science more exciting than academic research because so much academic research has gone off into the weeds and never came back. Corporate researchers do, at some point, have to justify to their managers why their research has some sort of practical utility even if it might be quite long term. Their findings are integrated into products that just won't work if the research is bogus. Academics are never tested in this way. With no ground truth to anchor them it's easy for entire fields of science to spin off into a groupthink dominated la-la land, in which pleasing your colleagues preconceptions is much more important to your success than actually establishing the truth. Outsiders are the only people who can bring them back.
That's why I want to see more oil companies funding research. I want to see GM food companies funding research into GM foods. I don't believe anymore that they're more biased than academics, they just have different biases, but in the end they're still writing papers and making claims that anyone should be able to check. Academics should not be destroying raw data and forcing outsiders to use millions of FOIA requests in order to check their work. They should be welcoming such scrutiny from whoever gives it and especially from outsiders who have different things to lose - just as they would expect corporate funded science to be subject to their own scrutiny in turn.
> I used to have near absolute faith in academic scientists, with much the same reasoning as you
If I had to summarise my position today from my particular life experience it would be "trust no one". Especially politics and commerce, but including lawyers, doctors and academia and the rest. It rather unavoidably goes with human endeavour.
Most of my career has been in corporations. That's included some time as founder.
That has not taught me to prefer commerce, but to deeply mistrust it. The non-technical CEO or manager who requires some negative activity, or to not mention that our product is entirely unsuitable for the large sales prospect. The IBM or HP consultant presented as field expert, costing hundreds per hour, that over lunch turns out spent only one week on a training course. Last week in fact. Their only exposure to that niche.
Elizabeth Holmes conducting a personal visibility campaign selling snake oil. Bayer and heroin - the non addictive pain killer. Asbestos - the universal and entirely safe substance.
I could go on all day with the fraudulent, but then there's the inconvenient.
Sure GM companies have great research into GM. Then solely publish the studies that favour continued sale of their product. Maybe there are 30x more studies in their filing cabinets that show up the major danger. They stay in the archive, never published. No one knows they exist. Mention them and you are in breach of your NDA and contract of employment.
With luck, like tobacco, they might leak over the coming decades. So we only ever find out the negative far, far too late. or after regulator (that USA has tried to mostly gut), or academic (that the entire world has pushed nearer and nearer commercial intent)
That's also circular. With far stronger incentives to fraud. Fraud, fighting to ruin something that works and partial information appears to be the regular, day to day normality of commerce.
Which is probably a fit place to mention economics. Until the US deconstructed Bretton Woods and abandoned the gold standard, kicking off the inflation of the seventies, the groupthink was Keynesian. Though just about no government was actually Keynesian, most having adopted the obvious bits and completely ignored the challenging bits - that were the parts intended to compensate for the excess.
Come 1980 and we've moving into the groupthink of Friedman and Hayek. Less economics more extremist politics with a veneer of economics - particularly Friedman. Once again, just about no government is properly Hayekian adopting the convenient "free money" parts. The fact that both were far into political, or specifically irrational phobia of Communism in every corner, opinion ruins their economic credentials.
So someone trying to get an economic education can only get the groupthink. Just about none of the economics, or far more dangerous, Philosophy, politics and economics (PPE) courses that all modern politicians seek present the other economic models.
So bullshit all around us, but I find from my career, hearing of everyone else's career, and all I have read and experienced that the strongest negative incentives - to scientific honesty - come from corporations. If we had some trustworthy way to require research be declared before start, and results published to all, you would probably be right. As they frequently do indeed do the best research. They just never publicise the negative.
I can't sum up every possible argument in a HN post, only show that there are problematic signs worthy of further and deeper inspection. That is, skepticism isn't some weird cranky idea. At least some of it is based, at heart, in the question of whether there are mistakes in algorithms or software. That's a totally normal question to ask about any data analysis.
For instance, let's take your audit point. You audit the data going back. In the USA this is somewhat possible. For the HadCRUT dataset it's not, because they lost/destroyed the original raw data, and their computations to clean it up are not reproducible, as the notes about "Harry's" attempts showed quite clearly. That is already a major problem; in effect we're being asked to take the correctness of CRU's data adjustments on faith.
So let's focus on the NOAA dataset. A large amount of it is synthetic, more and more over time. You say we can't just scream "fake data" and remove it, and I agree we can't because that would at this point erase most of the official US temperature record. But ... seriously? You don't find it at all concerning that in the past humanity was apparently capable of measuring temperature with thousands of weather stations across the USA, and now we've apparently lost that ability just when we need it most - forced to rely on 'estimates' generated by computer simulation? The foundation of science is fitting theories to match the data, but here, the data is being generated by the theories. I think anyone who cares about science should care at least about that.
Now, which adjustments are questionable and why - that's a great question and is pretty much where I'm up to in my own research. There are two main strands of alteration: TOB adjustment, and the PHA.
TOB is "time of observation bias". PHA is the pair-wise homogenisation algorithm. There are other adjustments made too: more and more algorithms have been added over time, but those seem to be the two big ones. PHA seems to be involved in the 'estimation' process to fill out or adjust the reported weather station readings.
This graph shows the affect of them split out:
https://realclimatescience.com/wp-content/uploads/2019/08/US...
As you can see the adjustments have little/no impact today. But they rewrite the past significantly: raw data pre-adjustment doesn't show any warming in the 20th century, the 1940s in particular were as hot as today. TOB adjustment was developed first and pushes down the temperatures in the past, then PHA+extras (what this graph labels "final") was developed later and pushes it down further.
This can be interpreted in a few different ways. One way is that climatologists found that in the past people were consistently bad at measuring temperature accurately, so they process the data to remove the failures and the data is now accurate. Another less charitable interpretation is that the algorithms have the effect of fitting data to match the theory, a form of scientific malpractice that is not unknown in history. To figure out what's going on requires much more analysis.
Note that nothing here requires an evil conspiracy. Climatologists were faced with a huge problem in the 1970s: basic chemical theory said CO2 is a greenhouse gas, we're burning lots of fossil fuels, so the world should be getting warmer. At least if you assume the climate is quite simple. But the raw data didn't show that. It was also messy, filled with glitches and could obviously benefit from some cleaning up before usage. So scientists started trying out various analyses and adjustments, but ultimately they're not comparing against a ground truth. They don't know what the data "should" be except through climate theory, so it would be very easy to end up trying lots of algorithms and ending up selecting the ones that result in "obviously correct" outcomes. Meanwhile the number of people involved in this is very small, they're all colleagues, their income depends on the belief their understanding of the climate is always growing, and it's unclear how you could ever resolve the apparent contradiction between chemical theory and raw thermometer data in a field like climatology. It's not like you can experiment on the Earth.
In other words, there's a big grey area between cleaning up the data and adjusting the data to fit the theory. Did they go into that zone, and if so, did they cross the invisible line? It's subjective, but not obviously impossible.
Anyway back to the analysis. So far I only found an analysis of TOB adjustment. It can be found in the second half of this rather long blog post:
https://realclimatescience.com/erasing-americas-hot-past/
Start reading from "I have new tools". The quick summary is: some weather stations used to be read in the morning and others the afternoon, and this was suspected to need fixing. The author analyses Minnesota because it has a balanced mix of AM/PM stations, and shows the PM stations are indeed consistently reporting about 0.5 C of difference to the AM stations. But then he analyses their coordinates and finds that this is because the PM stations are on average a degree of latitude further south, and that average temperature increases about 1C per degree of latitude, so you would expect such a difference based on location alone. In other words, correcting for location eliminates any difference that could be caused by TOB.
I'm still in the process of checking these claims. I didn't reproduce the graphs yet like I did for the claim of synthetic data. The claim 1 degree of latitude = 1 degree C of warmth appears backed up by this source:
https://www.physics.byu.edu/faculty/christensen/Physics%2013...
So it sounds plausible. If no TOB is observable in Minnesota, why did NOAA scientists conclude it was a necessary adjustment to make everywhere? It's also kind of suspect that this would matter, since random noise should get washed out by the averaging and long term trend analysis.
BTW, you ask do I accept that data has to be adjusted because stations moved. I certainly did at first, when I started looking into this. The deeper I look the more I'm starting to question this. Station moves and upgrades can certainly introduce artifacts into the reading of a single station, but, scientists only really care about global averages. For this to require correction at all would require the effect of moves to be highly correlated.
Final notes on the science.
what you very probably should never do unless your intent is to mislead, is simply present 100% raw data on a graph
That's not the science I was taught! One of the pillars of science is providing your raw data. If there is uncertainty in the measurement that's what error bars are for, and the various statistical techniques for handling uncertainty. Picking data points and excluding others that don't fit the theory is literally "bad science 101".
Also, if climatologists have such severe concerns over the integrity of their raw data that they need to interpolate or synthesise nearly half of all readings, they should be screaming at the top of their voice that nobody should do anything until the global weather station network is in a better shape and that should be their top funding priority. Are they doing this? If so I've never seen any reference to it. We should be seeing more measurements and better data over time, not a decline in the number of trustworthy weather stations (which is what's happening, hence the rise in synthetic data).