Re:After ‘The Biggest Loser,’ Their Bodies Fought to Regain ...
by arkades on Monday May 02, 2016 @11:58PM
(#11617808)
You confuse p-value with methodology. P-value is entirely a product of sample size and effect size. For a given effect size, the cheaper I want the study to be, the smaller I make the sample, and the less likely I am to reach a significant p-value.
The problem is when the methodology itself is broken. For instance, using case-control methodologies on things with small-to-moderate effect sizes. Past studies comparing such studies with eventual RCTs showed that the case control studies gave absolutely no indication of what the RCTs would show as a consensus: it was entirely random.
There is no amount of money, small or large, worth throwing at studies that shift our prior probability to posterior by 0 units.
They actually do. A crappy study with p=.70 isn't enough to justify changing the standard treatment for a disease but it is enough to justify doing the more comprehensive study to see if the result still holds.
We have a choice between doing ten comprehensive studies only to prove that nine of the things didn't work, or using the same resources to do a hundred crappy studies and then repeat the five with the most promising results using the more robust methodology and discover that three of them still hold.