News for hackers. Stuff that matters.
Re:CRU's "very artificial correction for decline" is dead code by Tichy on Saturday November 28, 2009 @10:33AM (#965205)
"that code was written to be used for some kind of presentation that was false. "

In other words, he doesn't know a damn thing. I can't help but feel that the anti-warming crowd (which includes ESR if I remember correctly, he wrote some long rant about peak oil?) is trying to blow a tiny mite up to be an elephant. It's typical revisionist behavior, just like the creationists who find a small bone that is not where biologists have predicted it would be, and they think they have refuted Darwin.

In any case, this all seems to be very ad hominem (revisionists trying to discredit scientists instead of providing data). Even if somewhere out there in the vast internet a false chart is floating around, it does not refute global warming. (Note: I don't claim that global warming is real - I don't know. I just don't like the style of this "mite-gate").

The way to resolve this seems to create some proper charts off proper data.

Re:CRU's "very artificial correction for decline" is dead code by yummyfajitas on Saturday November 28, 2009 @11:02AM (#965226)
(revisionists trying to discredit scientists instead of providing data)

If you claim P=NP, and I show a flaw in your proof, am I also required to provide a proof of my own? Of course not.

This is how science should work; both scientists, their competitors, and anyone else with a passing interest will carefully try to discredit existing work. In this case, it looks like a specific attempt to discredit it failed.

You are correct that creating proper charts off proper data would be the best solution. Unfortunately, much of the data has been "lost", making this impossible.

Re:CRU's "very artificial correction for decline" is dead code by ewjordan on Saturday November 28, 2009 @11:36AM (#965261)
I think the parent post was saying that the "revisionists" like to pick out weak points in one particular small result, and try to unfairly imply that all of the larger results in the field must be wrong, as well, without touching on any of the data or arguments in support of those results.

This is what happens in the evolution debate on a regular basis: someone comes up with some minor technical complaint about peppered moth observation, spends a long time debating it, and then declares that evolution is "in controversy" despite the fact that even if their complaints were valid, there are thousands of other pieces of evidence in favor of the larger theory.

I don't know enough about global warming to know if its critics are merely hacking away at a weak sapling to draw our attention away from the dense forest that they can't cut their way through. But I know that tactic has been applied in the past, so I don't think it's unreasonable to wonder.

Scientists tend to get caught up arguing over whatever particular claim is being criticized, but I think in this situation, it would serve the climate scientists well to point the public eye towards less vulnerable data and claims, if they're out there...

Re:CRU's "very artificial correction for decline" is dead code by Tichy on Saturday November 28, 2009 @12:04PM (#965284)
Sorry I missed how you mean a flaw in the proof. The thing about evolution theory is that evolution is a stochastic process. So there tend to be not precise predictions, just likelihoods. Also, it is being applied in thousands (or millions) of cases to explain complex systems. If sometimes somebody who applies it gets it wrong (like say some biologist trying to make sense of bone findings and creating a theory of the evolution of some species or other), it does not refute the whole of evolution theory. It only refutes that particular application.

It is like physics: millions of buildings have been built employing Newtons laws. Sometimes an architect gets it wrong and the building falls down. But that doesn't refute the laws of physics - it is just some guy or a team of builders making an error.

Re:CRU's "very artificial correction for decline" is dead code by yummyfajitas on Saturday November 28, 2009 @12:28PM (#965300)
I'm not disagreeing with that. AGW may be true while the corrections to this data series may be false, and this data series may be only a small piece to the puzzle.

However, fraud and errors are different. Science is (to a great extend) based on trust. When I referee a paper, I trust that Fig 1 is really generated by Algorithm 2 if the caption says so. If it turns out that Fig 1 is photoshopped, my report is meaningless. Peer review is meant to find mistakes and evaluate importance, not eliminate fraud.

If a scientist cheats, we must consider all their work to be flawed until they can prove otherwise (i.e., release all data, code, etc). We need to stop and scrutinize all the fruit of their poisoned tree.

AGW may be true, and the science may be solid. But until we have more openness, we will never really know.

Re:CRU's "very artificial correction for decline" is dead code by Retric on Saturday November 28, 2009 @08:13PM (#965758)
Science is (to a great extend) based on trust.

No, it's easier to assume trust, but science works best when there is almost zero trust in any single authority. This is why it's expected for you to present your methods, data, math, and then your conclusions. Ignoring malice a tiny mistake can drastically change an experiments outcome. So an experiment yet to be independently verified is next to worthless, but it's not sexy to work like that and it does not help your reputation.

Edit: IMO, what's missing from this discussion is the idea that independently verifying their results is a reasonable thing to do. This does not mean plugging their numbers into their code and looking at the results but starting from scratch using their methods and testing it yourself.

PS: One of the reasons that social science has such a poor reputation is the willingness to accept minuscule sample sizes as representing real research. A little less trust would go a long way to revitalizing those fields. Unfortunately neuroscience seems to be falling for this trap.

Re:CRU's "very artificial correction for decline" is dead code by yummyfajitas on Saturday November 28, 2009 @10:30PM (#965868)
It's true that in the platonic ideal, referees would not trust authors.

In reality, referees do trust that authors are making a good faith effort to present their results accurately (except for a little bit of cherrypicking and glossing over messy parts). As a referee, I'm simply not paid enough to search for fraud.

Re:CRU's "very artificial correction for decline" is dead code by Tichy on Sunday November 29, 2009 @07:53AM (#966293)
Well the anti-warming crowd does have enough of an incentive to search for fraud, I would expect?
Re:CRU's "very artificial correction for decline" is dead code by yummyfajitas on Sunday November 29, 2009 @09:47AM (#966356)
They have incentive, but not the means.

To search for mistakes, I skim a paper, think "yeah, that graph looks about right", [1] and give an opinion.

To search for fraud, I would need the entire source tree for the paper: data, programs, etc. I'll need to verify that the source code does what the paper says, and that the output of the program is really the source of the graphs. (This ignores the issue of fraudulent data, but that's really tricky to find.)

Since the data was destroyed (oops, I mean lost), this simply can't be done.

[1] I also check the proofs for mistakes, but that's irrelevant for this discussion.

Re:CRU's "very artificial correction for decline" is dead code by Tichy on Sunday November 29, 2009 @11:22AM (#966445)
Sorry, I just don't buy this. If there is no data, then the paper (which one? is there one?) is worthless. Academia might be fucked up, but it is not THAT fucked up. Otherwise I'd hand in my PhD thesis tomorrow, proving that I cured AIDS - I just lost the data. So I don't see why we even discuss this case. It reminds of a recent story where the cleaning lady threw away the rare mice shit some PhD had been collecting for years for his thesis. Funny story for the magazines, but nothing with any impact. No data, no paper, end of story.

Also I don't think this was the only data set in the entire world that relates to global warming. So it hardly invalidates all climate research if some random researcher formats his hard drive.

As for means, isn't the whole fossile fuel industry supposed to be behind the warming skeptics? They should have shitloads of money. In fact I wonder if by now they have their own research stations in the arctic, drilling for historic ice. I mean, they should? They really can not afford to buy their own thermometers?

I think they would probably have enough money to buy dozens of universities dedicated to climate research.

Re:CRU's "very artificial correction for decline" is dead code by yummyfajitas on Sunday November 29, 2009 @12:22PM (#966512)
The data is claimed to have previously existed, but has now been deleted (links at bottom of this comment). In general, academia is not that fucked up. Medical journals certainly won't let you get away with this.

Climate science appears to have a lower set of standards.

Some of the CRU emails criticize climate journals which were considering adopting a "release all data" policy, and argued for not submitting papers to those journals. Other CRU emails discuss deleting data to prevent Steve McIntyre from getting it, and conspire not to release the data to him or other skeptics. This is why climategate is such a big deal.

As for other data sets, there are a few others. However, the CRU is one of the major sources of paleoclimate data. They aren't the only game in the world, but they are a big game. Both their data, and various analysis which are based on their data, are now inherently suspicious.

To make an analogy, imagine that a major open source regex library (maybe PCRE) is completely tainted (e.g., a rootkit). How large an impact would that have on open source software?

http://www.timesonline.co.uk/tol/news/environment/article693...

http://news.ycombinator.com/item?id=966336

Re:CRU's "very artificial correction for decline" is dead code by Tichy on Saturday November 28, 2009 @09:22PM (#965813)
I certainly don't want to defend fraud. However, so far I have not seen the evidence that fraud is the right word for this mite-gate. I am sorry that I even spend so much time discussing this, but I am horrified at how fast many people seem to be jumping the gun here.

I see some emails that give a strange impression. How about investigating more details before calling it fraud? ESR can not even point to an actual application of the alleged corrections. I think for fraud we would need a little more.

Note I don't say it wasn't - I don't know anything about the people involved, their standing or their research. But I believe in the good old innocent until proven guilty. I think it is one of the cornerstones of civilization.

Re:CRU's "very artificial correction for decline" is dead code by Tichy on Saturday November 28, 2009 @11:28AM (#965248)
It seems I am missing the "trying to discredit" part, where is that from?

So far I also haven't seen any "flaws in proofs". I have only seen words like "trick, milk data" that can or can not mean bad things - take your pick depending on what you want to believe.

Re:CRU's "very artificial correction for decline" is dead code by DannoHung on Saturday November 28, 2009 @11:14AM (#965237)
Science isn't math.
Re:CRU's "very artificial correction for decline" is dead code by yummyfajitas on Saturday November 28, 2009 @12:10PM (#965288)
I don't recall claiming it was.

I simply used an example most people here would be familiar with; my broader point remains unchanged for data analysis. One can legitimately critique an analysis of data without providing new data or even performing a correct analysis.

Re:CRU's "very artificial correction for decline" is dead code by cwan on Saturday November 28, 2009 @10:44AM (#965211)
I agree - the remedy is transparency and to understand how, where and when these models are used - especially if they're being used to form significant public policy as is the case here. It seems reasonable to believe that the code was used for some purpose - even if it was dummy data - but clearly questions remain that should be asked and answered. What makes it even more troubling is the attitude of engagement of these scientists as if they are above being questioned when it is now revealed their emails suggest doubt, evasion and even suppression of valid but differing studies.

I've come to believe that AGW probably exists but am not married to the idea either - but nor am I convinced we need massive corrective policy versus adaptation (heck - I live in Canada after all, and after last winter we could use a little warming). That said, and I'm not saying you're doing this but many proponents of AGW are attempting to minimize the implications in the emails but their impact cannot and should not be denied: http://www.guardian.co.uk/environment/georgemonbiot/2009/nov...

Also - given the lack of transparency and the emails that have now come to light, the researchers most certainly open themselves up to questions that include whether or not the underlying data has been manipulated/adjusted such that the coding adjustments are no longer necessary http://wattsupwiththat.com/2009/11/26/mcintyre-data-from-the...

Re:CRU's "very artificial correction for decline" is dead code by Tichy on Saturday November 28, 2009 @11:25AM (#965246)
The guardian link seems very weak in content - trying to get a paper published is not fraud. Sorry, but all I see is the news trying to make a story.

The whattsupwiththat thing was discusses in another thread, again I don't think it is all it is cracked up to be. It is just anti-warming people jumping at a chance, addressing the dumb public who doesn't understand scientific jargon or processes.

If this leads to a better grasp of the public about this, it might at least be some benefit. I am not claiming that academia is perfect these days.

Re:CRU's "very artificial correction for decline" is dead code by CamperBob on Sunday November 29, 2009 @12:12AM (#966000)
The guardian link seems very weak in content - trying to get a paper published is not fraud.

It might as well be. It wouldn't be an exaggeration to point out that the consequences of basing political and economic decisions on the content of those papers will run into the trillions of dollars.

If there were ever a field in which it's important to avoid even the appearance of impropriety, climatology is it.

Re:CRU's "very artificial correction for decline" is dead code by Tichy on Sunday November 29, 2009 @04:52AM (#966175)
I'm sorry, but if such decisions are made without even double checking the results, then maybe it is not even the scientists to blame.

Also, this is pure speculation. From ESR's article I gather that it is not even known how and when the data was used. They could not yet point to an actual published article.

And, who are these people, are they like the most famous climate researchers ever, the Einsteins of climate research? Are they the only climate researchers on the planet? I can't imagine that they have that much influence.

Re:CRU's "very artificial correction for decline" is dead code by gaius on Saturday November 28, 2009 @11:09AM (#965229)
revisionists trying to discredit scientists instead of providing data

OK, but kinda the point of this is that these scientists themselves attempted to discredit other scientists who disagreed with them, rather than refuting them through data. That lays them open to ad hominem attack.

Re:CRU's "very artificial correction for decline" is dead code by Tichy on Saturday November 28, 2009 @11:26AM (#965247)
When did that happen? I did not read that in the emails that were floating around here.
Re:CRU's "very artificial correction for decline" is dead code by gaius on Saturday November 28, 2009 @12:16PM (#965291)
http://online.wsj.com/article/SB125883405294859215.html

Phil Jones, the director of the East Anglia climate center, suggested to climate scientist Michael Mann of Penn State University that skeptics' research was unwelcome: We "will keep them out somehow -- even if we have to redefine what the peer-review literature is!"

Re:CRU's "very artificial correction for decline" is dead code by Tichy on Sunday November 29, 2009 @05:06AM (#966186)
What does that even mean? This is all so much speculation.
Re:CRU's "very artificial correction for decline" is dead code by CamperBob on Saturday November 28, 2009 @01:17PM (#965344)
In any case, this all seems to be very ad hominem (revisionists trying to discredit scientists instead of providing data)

Yep. That's a great summary of the contents of the emails.

Re:CRU's "very artificial correction for decline" is dead code by Tichy on Sunday November 29, 2009 @07:54AM (#966294)
How so - there were not even any names of "enemy scientists"? Nor were there any references to claims of anti-warming people, or were there? If not, how could there be any ad hominen?