> is it really so far-fetched to argue that perhaps climate scientists are as biased as researchers in areas such as medicine and linguistics?

No, it isn't, unless you believe that climate scientists are a definitionally more moral group than others. The incentives remain the same, in all branches of science, so manipulation is likely the same in all branches.

Physicists have a better understanding of statistics and better data to work with than pyschologists. There is a very clear causal explanation for why greenhouse gases do what they do. With that said, there is not enough discussion about the cost and effectiveness of proposed solutions to global warming - a lot of the "solutions" are more feel-good measures, which are not worth the cost for what they do. Global warming is a global problem - if the US stopped all of its contributions to climate change the earth would keep warming up.

As a physics grad student I have seen plenty of "negative" results published, and the standard of positive results for instance 3 or even 5 sigma is a much tighter standard than p > 0.05. Science is a big field, the problems in one domain do not necessarily translate into all domains. However there are other fundamental problems with physics as a field.

The question in climate change is not direction, but magnitude and variance. While most of physics does use 3-5 sigma tests due to a large number of experiments, climate science can not do the same.

Furthermore, we have significant evidence that climate scientists are motivated to protect their work against critics - just read the climategate emails. That's not how science should be done, and goes well beyond the replicability issues discussed here.

At the risk of further derailing this thread, there was nothing wrong with the content of the climategate emails, other than poor choice of words - a poor choice of words that were all too easy to take out of context.

Someone said something along the lines of "I'm not giving you the data because you'll try to poke holes in it" as a FOIA request answer.

How is this OK?

That doesn't relate to the actual content of the emails or data. Although I agree that they dealed with the FOI requests poorly.

Really? There is 'literally nothing wrong' with stonewalling FOI requests, stacking peer review to prevent opposing scientists from publishing, or splicing temperature data onto the end of a proxy chart to hide the fact the proxy is diverging?

Ok then.

>There is a very clear causal explanation for why greenhouse gases do what they do

I'm sorry, but it doesn't work that way. We have very clear chemistry/physics on the composition of food, how it is digested, etc. But nutrition science is still an embarrassing shitshow.

We have clear physics on how neurons fire, etc. but neuroscience is still in its infancy and psychology... well, it is psychology.

Earth's climate is a large scale multi-variable control system with thousands of feedback loops. A good understanding of the physics that drive a single forcing doesn't really tell us shit I'm afraid. It is just as open to manipulation as these other fields.

This sentiment is not correct - causal explanations do matter. The statistical evidence for global warming is not particularly strong, as temperature data is very noisy. If we didn't have a causal explanation there would not be a scientific consensus behind climate change. That adding greenhouse gases to the atmosphere changes the climate is more like gravity - we have clear physics on why things fall when you drop them, and if we somehow were adding mass to the center of the earth we know what effect this would have.

More generally you are correct, we don't know exactly how much the earth will warm or if there are complicated feedback mechanisms in place that could cause this warming to speed up or reverse course. We can't even reliably say that next year will be hotter than this year (actually it probably will be cooler because this year has been unusually hot).

Indeed, but it is really only the system response we are interested in. As you admit, a clear causal explanation for how a single input forcing is increasing doesn't automatically get us to a position where we can predict the overall system response. Or predict whether a given system response will have some positive or negative second order effect.

We have a clear causal physical explanation that eating fat should make you fatter right? Are you willing to say that? Or would you qualify yourself - 'eating more fat will make you fatter in the absence of negative feedbacks (perhaps the fat reduces your appetite more than carbs or protein?) and assuming that all other inputs (exercise level etc) remain constant'.

Really, all we have is how well a model predicts the future. there's a fairly sucky model from the 80's, [1]. Maybe it's totally wrong. Maybe it's just overfitting. But it's got a good track record, if underpredictive. I'm all for the idea that co2 doesn't effect temperature. I just haven't seen any models that show that. Furthermore, i haven't seen anything with any sort of a track record. I'm pretty sure Hansen's paper is pretty much the root of all modern climate modeling. I'm totally willing to stipulate that the whole field is wildly off track, but there's no evidence of that. The only thing we have is 35 year old models that seem to work.

I mean, more specifically, aether was flat out wrong. it still had predictive power. Someday we'll have something more refined, or perhaps completely refuted.

[1] http://www.realclimate.org/index.php/archives/2012/04/evalua...

Derailing the thread further, I don't understand why people say "aether was flat out wrong". Is it because they don't know it was just a name change, and we're now calling it "vacuum"? [1]

[1] https://en.wikipedia.org/wiki/Aether_theories

See for example,

The modern concept of the vacuum of space, confirmed every day by experiment, is a relativistic ether. But we do not call it this because it is taboo.

Interesting.

Compare the chart in the linked article with Hansen's own assessment that he put out in 2005 (third page): http://www.columbia.edu/~jeh1/2005/Crichton_20050927.pdf

How do you reconcile those two? A lot seems to come down to centering decisions?

Scenario A I believe is closest to the emissions path we are on.

My guess is, the pink band is current (er, 2011 versions) of what the global average temperature was. Which makes the 60's even cooler (heh). As with all things, it depends on how you measure.

global mean global giss is here - http://data.giss.nasa.gov/gistemp/graphs_v3/

So anyway, i'd agree the centering of zero sure could change things, but it sure seems like predictive power in the rates of change. Like, temperature isn't stable, it's not going down, it's going up and it looks like it's going up at about that rate.

edit

but that sort of goes back to the original point, all we really have is models. you held up physics as an example, but if we look at something like the gravitational constant, things are really screwy. As far as i can tell, 3 really first rate teams came up with 3 different answers - not even overlapping in the error bands. big G is obviously helpful, but it must be more complicated than we understand right now.

I dunno. I kind of like the european model for chemical handling. Super toxic chemicals in tiny quantities aren't that big of a deal, maybe 50 people die, so it's not heavily regulated. Mildly toxic chemicals in large quantities, same deal. Maybe 50 people die. Large quantities of toxins are regulated in proportion to the risk. The point is, balancing the risk against the best current understanding really seems like the best that can be done.

Sure, but a pretty high percentage of the US believes anthropogenic climate change is not real. They think it's either a misinterpretation, or an explicit hoax. Most of the Republican party denies climate change, or at least denies that human activity is playing a significant role in it.

We can't have reasonable debates on cost-effectiveness when so many people deny the fact that there is even a problem at all.

This is orthogonal to my point, which is that these incentives are present in all branches of science, and are likely impacting what is published similarly.

I'd be interested in reading any thoughts you have to share regarding other fundamental issues in the field of physics.

I can only speak to particle physics, but the main issue is that we can't study the truly interesting problems. Theories such as string theory are basically beyond the reach of experiments. Our current models work very well to describe the universe, but we haven't made that much serious fundamental progress since the 1970s. It takes decades to find new particles which we know must exist (top quark in 1995, Higgs in 2012). This will probably become even more true after the next few years after the LHC collects data at 13 TeV, although of course I could be wrong and something could be found. To study new physics you have to go up an order of magnitude in energy or luminosity, and this scales worse than linearly with cost, so it isn't feasible. Of course it is possible that new technologies emerge, but this isn't a sure thing.

The other problem with physics is that it is really hard to become a professor, and the field forces 90% of bright, dedicated and talented people to go into industry because there is a lack of jobs in physics. We really need permanent positions at labs outside of academia.

>Physicists have a better understanding of statistics and better data to work with than pyschologists.

They certainly have better data to work with in most cases, but what makes you think that physicists understand statistics better than psychologists? Is this just the physics superiority complex?

No. Physicists just take a lot more math classes (including statistics and data analysis) during their undergrad/graduate studies than biologists or psychologists do.

Physics PhDs will have taken more math courses than Psychology PhDs on average, but I am pretty skeptical that they have taken more statistics courses. I would like to see the evidence for that if you have it.

Sorry, my point was simply that the incentives discussed in the post aren't unique to the field. They play a role in all academic literature, the best way to deal with it is to be consciously aware and take advantage of what the internet has allowed in terms of non-traditional distribution.

Another point that is briefly touched upon is the availability of data. What good is a study I can't verify? I wish there was a tag that could be applied to published papers along the lines of "the original data is not available or the author is unwilling to share it, treat this very septically". If you ever feel like having a sad couple of weeks, pick 50 empirical papers and send the authors emails along the lines of "I just read your paper X, very interesting. Would you kindly provide me with access to the data so that I can Y".

Call me naive but reproducibility is one of the hallmarks of a good scientific publication. I'm also fairly skeptical of studies where the used questionnaires are not openly available...the list goes on (I'd also argue that closed source software should be avoided because "we plugged our data into the not checkable algorithm X and got result Y which we interpreted according to what the authors of the software say we should interpret it as" isn't great either)

s/septically/sceptically/ (probably :-))

And preparing data for publication is lots of work [1] that unfortunately doesn't 'pay' in today's scientific culture.mit even runs the risk of others 'mining' publications from your data before you have the chance to do so.

The best reply to questions you can expect is "that's a nice follow-up idea. Let's work on it together". Risk there is that the scientists may be working in some other direction that they find more promising.

[1] for example, you have 100 sets of data, but only used 78 in _this_ publication for various reasons (chemical used was applied at incorrect time, for too long, was out of date, subject was too old, had an existing condition, etc). You explain that in the publication in a small paragraph. In your dataset, you probably did it by copying the dataset and removing the ones not suitable for the task at hand. Did you keep that selection after the paper was published? Probably not. Even if you did, sending the selected data set out isn't sufficient. Others might want to see all data, so that they can verify the correctness of your selection.

Source control systems could help here, but as I said, providing data unfortunately career-wise doesn't pay at the moment for scientists.

> and get some sort of crowd-sourced consensus from researchers with credible reputations.

Would be nice, but where would the money come from? Even if you live in a country with high taxes that fund ridiculous projects, they are still often accountable to the public. Would the public want to fund new research or try to replicate or disprove other studies? The only way to make it work would be to build-in a certain percentage of funding for such a purpose in reaction to some gross oversight, so you could get the public behind it, or perhaps get some wealthy donors behind the cause, but I think you'll have a difficult time.

The better option might be having research students be required to replicate at least one study that hasn't been replicated in addition to doing their own research. That way you get some free labor!

Depends. I would like to think that "the public" would notice all those silly cancer patients showing up to clinic (don't they know that cancer is cured twice a week at every major hype factory, err, university?) but, no.

Pie in the sky it is, then! If only a field existed that attempted to quantify the expected variation from a given size and design of experiment.

That field would be of great use to major nation states. So much so, you might call it statistics. The practitioners would probably be big party poopers, always referring to weird concepts like "regression to the mean", "power", "effect size" and other unexciting yet critical details. I imagine there would be funny videos on YouTube about these people, and "real" researchers would happily ignore them while they jerk each other off in P01 study sections.

Anyways, let's keep science the way it is. There's no way the process or its results could possibly be improved!

> Anyways, let's keep science the way it is. There's no way the process or its results could possibly be improved!

Let it be know that this is sarcasm, for those that have trouble determining that.

Note: I was being completely serious about having graduate students being required to replicate studies that have not been replicated, or to disprove them. That was not sarcasm.

Graduate students are effectively required to replicate studies already. They just don't get any credit for the work when it turns out that the flashy paper didn't replicate, and the journal editors hate to look foolish, so it's exceedingly rare for failures to replicate to see a journal. So the next graduate student to work on the technique also gets to waste months or years on it.

Really a splendid system, isn't it? Sometimes (rarely) I sympathize with industry types who want grant claw-back processes. Then I remember that those are the same crooks that pushed Vioxx and 510k "equivalent" medical devices. And the sympathy evaporates, because they're even worse.

> There needs to be a better, robust method to publish every relevant study that is not utter crank, and get some sort of crowd-sourced consensus from researchers with credible reputations.

Isn't that the Internet?

I guess the problem on the Internet is that you also get every irrelevant study that is utter crank, and a crowd-sourced consensus from laypeople with no credibility whatsoever.

Peer review is boosting with three weak learners. If you think that has much credibility (after STAP, arsenic life, LaCour, etc) you clearly haven't been paying attention.

Nb. I review for various journals, but the process is far from foolproof. I do the best I can, but editors are free to override us in the interest of "impact". At least with preprints and PubMedCommons anyone with a cogent rebuttal can present it. Cell Press must hate that...

Curious question -- when you or someone else does the peer review, is it standard practice to actually double-check the math behind the paper, or audit any of the data?

No. At least not in neuroscience/psychology.

I've never seen a review request that was accompanied by raw data--you typically get an unformatted version of the manuscript, along with the tables and figures. That's it.

The reviewers can comment on anything, but they tend to be pretty conceptual. For example, one might say that the manuscript claims X, but the authors need to rule out competing hypothesis Y. Good reviewers will either suggest ways to do that (e.g., by doing a specific control experiment or citing a paper that rules Y out). They might ask you to comment on why your data claims A, but some previously published work indicates that !A.

To the extent that statistics get reviewed (not enough), reviewers typically comment whether they think the methods are suitable for the specific application or whether their output is being interpreted correctly. However, it's exceedingly unlikely that someone will actually check whether a t-test on the data in Table #2 actually gives the p-value printed at the bottom (or whatever).

Agreed.

I see it as both a cultural thing as well as practical thing (perhaps what initially lead to it becoming cultural). The reason for the second part is because most analyses are not easily reproducible--they exist as a smattering of data files and analysis scripts that coalesce into a magical, undocumented pipeline that spits out statistics and figures. Reproducibility has received more attention lately but it's still an uphill battle against academics' frantic publishing schedule, lack of familiarity with their software, and general lack of incentive for others to replicate the results (for some, a reverse incentive exists).

Thanks -- that is interesting to hear.

For me, yes. For others, no idea.

I run code, I check derivations, and I usually sign my name.

Science is ultimately self-correcting, but that can take decades. Sometimes you have to wait for distinguished asshole scientist to retire or die before happens. For example the Harvard Geology Department was the laughingstock of the world for taking so long to accept the thoery of plate tectonics. But eventually it did after rotating in a new generation of modern faculty.

Every few days I read that science is broken or losing credibility. And yet somehow the torrent of new technology, medicine and understanding of the world just keeps coming, and often makes my life better. So maybe not completely broken.

CRISPR, Higgs' boson, gravity waves, deep networks, self-driving cars. Not broken.

It's not about technology. Bullshit science affects policies. Also you don't see what did not get discovered/invented because of people having wasted time trying to build on bad theories.

Also, nobody's saying it's completely broken.

Those are major advances that happen very very rarely. The bulk of science is small incremental steps in science. It is extremely frustrating not being able to trust the results published even in high impact journals. This is a particularly glaring problem for theoreticists/modelers. Each theory/model must rely on potentially hundreds of studies, and it's really hard to assign a confidence level to each of them.

CRISPR is a genuinely novel scientific advance. The other items in your list are not.

The Higgs mechanism was concocted half a century ago; gravity waves are even older (general relativity is more than a century old). Their experimental confirmations were engineering feats.

Neural networks are from the 50s. Add fast computers and you get deep ones. Put them in cars and they learn to drive themselves.

It may take a toll of the social sciences and biology. It's not a big problem in physics or chemistry.

Oh, "Hollywood" chemistry is pretty fucking bad, but for different reasons. I miss being a real scientist sometimes.