Right. And then you'd have to use proper statistical reasoning for that state of affairs. Which nobody ever does, cause they're not statisticians and it's complicated and it would reduce the chance of 'statistical significance'.

So they just use a standard calculation of statistical significance -- which is based on the assumption that you have picked a single hypothesis in advance and then done your test. So it's completely invalid to use it how everyone typically does.

Imagine you flip a coin 50 times. Then you see, okay, did I ever get 10 heads in a row? Nope? Okay, how about 5 heads followed by 5 tails? Nope. Okay.... try a couple dozen other things, oh, look, I got exactly 3 tails followed by exactly 3 heads followed by exactly 3 tails again! Let's run my test of statistical significance to see if that was just chance, or is likely significant -- oh hey, it's significant, this is likely a magic coin not random at all!

Nope. If you test everything you can think of, _something_ will come up as 'statistically significant', but it's not really, those tests of statistical significance -- which calculate how likely it is the results you got happened by random chance happenstance vs an actual correlation likely to be repeatable -- are no longer valid if you go hunting for significance like that.

Not just could, would. Choosing your hypothesis after you run the experiment is (or at least should be) a cardinal sin in science for good reason. At the standard p-value cutoff of .05, even when there's absolutely no effect going on the probability of getting a spurious positive result when you do n comparisons is equal to 1 - (.95^n).

So that 5% chance of a type I error if you only look at one test statistic jumps to 40% if you look at ten, and to 72% if you look at 25.

Here's a nice piece of gonzo journalism that deals with this issue: http://io9.com/i-fooled-millions-into-thinking-chocolate-hel...

In other words, I run a study to determine if jellybeans cause acne[1]. The result is inconclusive (p < .05). Now--after the results are collected--I wish to check the correlation between color and acne. There are 20 colors. Would it be statistically sound to "correct" for my cherrypicking by setting p = 0.0025? That would result in

    1 - (0.9975^20) ≈ 0.049

It's no panacea. When you use it you reduce your statistical power - in other words, your likelihood of detecting genuine effects goes down.

No. For one thing, how do you actually know n? If in your analysis you can make c independent binary choices, n would be on the order of 2^c. For most reasonable sequence of choices, n would be impractically large. And if you think you know n, how would you convince everyone else that your value of n is reasonable/trustworthy?

The xkcd case (which is solved by applying the Bonferroni correction) is special because there's only one test run in 20 experiments, so the correction is more straightforward.

There is a good paper about this garden of forking paths by Gelman and Loken [1].

[1] (pdf) http://www.stat.columbia.edu/~gelman/research/unpublished/p_...

That's only the probability of getting a positive result by chance; the total probability of getting a positive result even if there is no effect is much higher.

C.f. http://blog.minitab.com/blog/adventures-in-statistics/how-to...

Choosing a hypothesis after the experiment is run is perfectly valid as long as your experiment is valid for that hypothesis. Besides, you would always run a new experiment again anyway.

In exploratory research you collect a bunch of data, and then mine it for interesting associations that might merit further study. It's an essential part of the scientific process, but anything you find from doing it needs to be treated as extremely tentative because it's liable to produce spurious results at least as often as it finds genuine effects.

But that's not what this paper's talking about. It's talking about experiments that are being used to support the approval of new treatments and drugs. That's confirmatory research. In that realm you absolutely must paint the target on the wall before you throw your darts.

If the experiment is constructed correctly, it can support validation of multiple hypotheses.

I would completely believe that most experimentation being performed currently is not structured to make further exploitation possible.

Could you provide an example of an experimental structure that is immune to the statistics being discussed?

It should read:

I would completely believe that most experimentation being performed currently is not structured to make further exploitation, of the type desired/wished, possible.

There are almost certainly facts available that are not discovered/discussed from past experiments. Many of them are likely trivial and/or not what researchers would wish or hope that their data could tell them. However, they can still be mined.

In any case though, you would still re-run experiments to further validate/reject the hypotheses. That is simply basic science.

Are you claiming that this statement is untrue, or irrelevant?

> Besides, you would always run a new experiment again anyway.

Then surely the initial experiment doesn't count. Otherwise, why are you re-running it?

Every experiment will support analysis through a set of hypotheses. Just because you didn't select all of those hypotheses before the experiment ran doesn't mean you can't select it after the experiment.

Imagine that an experiment has been run, but you do not know the results (or even what was done). Now you select a hypothesis, if the experiment required to validate the hypothesis is the same as what was run previously, you can now look at and use the results.

A hypothesis is like running a query against a database. Many queries are valid, even though the data may not have changed.

>Otherwise, why are you re-running it?

Science requires it. Doctrine from one-off experimentation is religion (hard to dump).

A lot of science is about serendipity.

Yes, that might happen. The more likely outcome though is that we are saved from a lot of drugs that don't work any better than chance (or even worse).