News for hackers. Stuff that matters.
Re:Registered clinical trials make positive findings vanish by function_seven on Tuesday August 18, 2015 @05:53PM (#10082473)
I have a question. Would I be on firm statistical footing if I started looking for effects after the study, as long as I choose a p-value such that (1-((1-p)^n) < 0.05?

In other words, I run a study to determine if jellybeans cause acne[1]. The result is inconclusive (p < .05). Now--after the results are collected--I wish to check the correlation between color and acne. There are 20 colors. Would it be statistically sound to "correct" for my cherrypicking by setting p = 0.0025? That would result in

    1 - (0.9975^20) ≈ 0.049
[1] https://xkcd.com/882/
Re:Registered clinical trials make positive findings vanish by bunderbunder on Tuesday August 18, 2015 @06:44PM (#10082699)
It sounds like what you're describing is roughly the same as the Bonferroni correction: https://en.wikipedia.org/wiki/Bonferroni_correction

It's no panacea. When you use it you reduce your statistical power - in other words, your likelihood of detecting genuine effects goes down.

Re:Registered clinical trials make positive findings vanish by conistonwater on Tuesday August 18, 2015 @06:32PM (#10082649)
> Would I be on firm statistical footing if I started looking for effects after the study, as long as I choose a p-value such that (1-((1-p)^n) < 0.05?

No. For one thing, how do you actually know n? If in your analysis you can make c independent binary choices, n would be on the order of 2^c. For most reasonable sequence of choices, n would be impractically large. And if you think you know n, how would you convince everyone else that your value of n is reasonable/trustworthy?

The xkcd case (which is solved by applying the Bonferroni correction) is special because there's only one test run in 20 experiments, so the correction is more straightforward.

There is a good paper about this garden of forking paths by Gelman and Loken [1].

[1] (pdf) http://www.stat.columbia.edu/~gelman/research/unpublished/p_...