Re:Caffeine at night does more than keep you up late
by bildung on Friday September 18, 2015 @02:42PM
(#10241219)
> No, this is wrong. With small sample sizes you may get a statistically significant result, but it still might not be a real result and might not be reproducible. This is a major issue in science today and why a lot of studies can't be replicated.
Reproducability indeed is a major problem, but looking at statistical significance alone isn't the cure (especially if applied a posterior).
We should rather look at effect sizes and robust study designs.
In fact, modern studies aiming for causality often calculate the population size needed for statistical significance beforehand. It's a standard formula in most textbooks. You only need the expected effect size and then can calculate the population needed to guarantee significance.
Reproducability indeed is a major problem, but looking at statistical significance alone isn't the cure (especially if applied a posterior).
We should rather look at effect sizes and robust study designs.
In fact, modern studies aiming for causality often calculate the population size needed for statistical significance beforehand. It's a standard formula in most textbooks. You only need the expected effect size and then can calculate the population needed to guarantee significance.