Re:Most published research results are false
by Lewisham on Friday February 11, 2011 @07:13PM
(#2208770)
From what I've seen, I wouldn't say CS is immune to this at all. CS seems to only be accepting papers that illustrate some big win.
In my short PhD career, I've an article in the drawer where the result was inconclusive. This result, in and of itself, was fairly valuable. It spoke much about all the ways things went wrong. However, wrong was not acceptable for publication, it had to be right. Much hand-wringing was done to identify the bits that could be pulled out from it, polished, and then submitted (I was not especially pleased about this). The paper was, rightly, rejected, but all the valuable findings are still in my drawer. The pressure to produce Good findings rather than Bad findings does not meet my expectation of scientific pursuit.
I took great joy in seeing a history of science exhibition recently, and reading the notebooks of people like Newton and Darwin, who they themselves spend a lot of time writing out all the things that went wrong, all the concerns they have about their "results", and general humbleness. It speaks volumes that the greatest scientific minds that have ever lived have put more caveats and concerns in their published work than 98% of what is published today does. To get published, it seems you have to be 110% sure that your work is 120% amazing. I don't think this is a good state of affairs.
Re:Most published research results are false
by johndcook on Friday February 11, 2011 @04:11PM
(#2208150)
Math and CS are better off than experimental sciences. Theoretical paper often contain errors, but these don't usually invalidate the conclusions.
In experimental sciences, hard sciences like physics do have a lower error rate than sciences that rely heavily on statistics. But that's changing. See the second page of this article: http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWMode... The author argues that "Nature has gotten more tight-fisted with modern physicists" and that physics is now more dependent on statistics.
Re:Most published research results are false
by logjam on Friday February 11, 2011 @04:17PM
(#2208167)
> Note well. This is for clinical research.
My own experience is that there is much constant questioning about published clinical research.
In addition, decision-making of clinicians based on clinical research is tempered with other considerations, sometimes quite complex. Stephen Paulker commented in response to the original paper:
"From the perspective of an epidemiologist or a statistician, the relevant question is whether the study's hypothesis is true - i.e., is the probability of H1 greater than 0.5. For clinicians and their patients, the relevant question is whether a particular strategy should be followed in an individual patient or a subset of similar patients. That decision (or recommendation to the patient) will depend on the pre-study likelihood of benefit in that patient and on the relative magnitude of benefits and risks of that strategy, if the diagnosis in that patient is uncertain. For many such decisions, the "more likely true than false" criterion may not be the best decision rule. For serious diseases and treatments of only modest risk, post-study probabilities of considerably less than 0.5 may be sufficient to justify treatment."
Re:Most published research results are false
by arctangent on Friday February 11, 2011 @06:49PM
(#2208701)
I always found statistics deeply disturbing when I was studying it at school.
You can establish a null hypothesis and then test at some confidence interval (say, 99%) and find that your null hypothesis still holds.
But, being somewhat ingenious, you may decide to lower the confidence interval (to, say, 98%) and find that your experimental evidence is now significant enough for you to reject your null hypothesis and accept your alternative hypothesis.
Re:Most published research results are false
by brent on Friday February 11, 2011 @08:08PM
(#2208911)
This is why different communities accept p-values at certain levels (typically <0.05). You have to live with some amount of uncertainty when the data generating mechanism is random... that is unfortunately the nature of the beast.
Re:Most published research results are false
by arctangent on Friday February 11, 2011 @10:37PM
(#2209200)
I understand this. But let's say I ran some experiments and collected some data in an attempt to disprove theory X. What does it mean for me to say that at the 99% confidence level X is still true, but at the 98% confidence level it is not true? I just find it a bit spooky, is all.
Re:Most published research results are false
by Read_the_Genes on Friday February 11, 2011 @05:28PM
(#2208432)
Most studies use raw p-values. This, coupled with "exploratory analyses" which are not mentioned in the research, leads to a high false discovery rate.
Alternatives include using simulations to estimate false discovery rates (this can be done analytically for some problems). Bayesian frameworks can also be applied, depending on the amount and accuracy of prior knowledge of the system under study.
That said, I see nothing wrong with publishing results with nominal p-values, as long as the researchers indicate the weakness in their results. Meta-analyses can always come back later and use the results they publish.
What annoys me is seeing 50 or so statistical tests done, and then researchers stating they have found something when 1 of those tests shows a p-value of 0.05. Just using a Bonferroni correction, the simplest of all corrections, would demonstrate that findings like this are not significant.
Re:Most published research results are false
by amalcon on Friday February 11, 2011 @05:38PM
(#2208474)
The problem isn't with the use of p-values, per se: the problem is that the human brain tends to distort probabilities on either end of the spectrum. People see that there's only a 5% chance that something could have happened by chance, and they put that in the "Not possible" bucket.
It's a fundamental weakness of the human brain. There's not a whole lot we can do about it.
Re:Most published research results are false
by apl on Friday February 11, 2011 @04:50PM
(#2208298)
Hypothesis testing isn't the fundamental problem, it's how a given p-value is perceived. Getting a so-called significant result simply shouldn't be confused with conclusive evidence for the hypothesis.
This says nothing about C.S., Math, Physics, etc...
It's harder to publish a big result in Physics, for example, and have it stand for long without being questioned. Just ask Jan Hendrik Schön.
In my short PhD career, I've an article in the drawer where the result was inconclusive. This result, in and of itself, was fairly valuable. It spoke much about all the ways things went wrong. However, wrong was not acceptable for publication, it had to be right. Much hand-wringing was done to identify the bits that could be pulled out from it, polished, and then submitted (I was not especially pleased about this). The paper was, rightly, rejected, but all the valuable findings are still in my drawer. The pressure to produce Good findings rather than Bad findings does not meet my expectation of scientific pursuit.
I took great joy in seeing a history of science exhibition recently, and reading the notebooks of people like Newton and Darwin, who they themselves spend a lot of time writing out all the things that went wrong, all the concerns they have about their "results", and general humbleness. It speaks volumes that the greatest scientific minds that have ever lived have put more caveats and concerns in their published work than 98% of what is published today does. To get published, it seems you have to be 110% sure that your work is 120% amazing. I don't think this is a good state of affairs.
In experimental sciences, hard sciences like physics do have a lower error rate than sciences that rely heavily on statistics. But that's changing. See the second page of this article: http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWMode... The author argues that "Nature has gotten more tight-fisted with modern physicists" and that physics is now more dependent on statistics.
My own experience is that there is much constant questioning about published clinical research.
In addition, decision-making of clinicians based on clinical research is tempered with other considerations, sometimes quite complex. Stephen Paulker commented in response to the original paper:
"From the perspective of an epidemiologist or a statistician, the relevant question is whether the study's hypothesis is true - i.e., is the probability of H1 greater than 0.5. For clinicians and their patients, the relevant question is whether a particular strategy should be followed in an individual patient or a subset of similar patients. That decision (or recommendation to the patient) will depend on the pre-study likelihood of benefit in that patient and on the relative magnitude of benefits and risks of that strategy, if the diagnosis in that patient is uncertain. For many such decisions, the "more likely true than false" criterion may not be the best decision rule. For serious diseases and treatments of only modest risk, post-study probabilities of considerably less than 0.5 may be sufficient to justify treatment."
You can establish a null hypothesis and then test at some confidence interval (say, 99%) and find that your null hypothesis still holds.
But, being somewhat ingenious, you may decide to lower the confidence interval (to, say, 98%) and find that your experimental evidence is now significant enough for you to reject your null hypothesis and accept your alternative hypothesis.
Lies, damned lies, and statistics, indeed.
Alternatives include using simulations to estimate false discovery rates (this can be done analytically for some problems). Bayesian frameworks can also be applied, depending on the amount and accuracy of prior knowledge of the system under study.
That said, I see nothing wrong with publishing results with nominal p-values, as long as the researchers indicate the weakness in their results. Meta-analyses can always come back later and use the results they publish.
What annoys me is seeing 50 or so statistical tests done, and then researchers stating they have found something when 1 of those tests shows a p-value of 0.05. Just using a Bonferroni correction, the simplest of all corrections, would demonstrate that findings like this are not significant.
It's a fundamental weakness of the human brain. There's not a whole lot we can do about it.