Or rather, when you have people who have a good idea what they're doing, they will do a good job of getting the results they were looking for. And maybe that is why statistics are so popular in political debates.
The problem is not that people are not aware of the issue of testing multiple hypotheses. The problem is that (1) it's hard to say exactly what your hypothesis is before you've even looked at the data, and (2) it's hard to determine if people are choosing parameters for p-hacking or simply making choices based on their best judgement.
>Furthermore, p-values were designed to deal with experimental data. If you're doing an observational study, perhaps you should use statistical tools designed for that purpose.
This is simply wrong. p-values are equally relevant in both cases. E.g. I can use p-values to reject the hypothesis that consuming saturated fats is uncorrelated with weight gain, amongst the general population. It sounds like you are reaching beyond your actual expertise in statistics.
Here's what doesn't add up for me: this implies that for many configurations, p-values are invalid. But what was it about preselecting the experimental conditions that really makes them any different? What makes initially chosen hypotheses of higher quality than iteratively discovered, "p-hacked" hypotheses?
The need for careful selection of a limited set of variable combinations in advance seems symptomatic that the test being employed is not robust. I'm not convinced that even limited application of p-value significance testing is actually valid.
Imagine you are a scientist and want to find out whether the hypothesis that men are on average taller than women is true. If you could just exactly measure and take the average of the entire male and female populations you wouldn't need a hypothesis test.
Since you can't, you can do an experiment where you take a random sample of men and women. Now, you can do the average in the same way, but you need something to help figure out whether to trust the results. That's where the p-value comes in. The reason you need to be careful to select hypotheses in advance is because in order for statistics to help you account for error you need the noise in the data to be uncorrelated with the result you are trying to assess---which it won't be if you chose the result because it was the one that looked best after you account for the noise.
Bob has a class of 100 undergraduate students. He tells them to go and run one experiment each as part of their final year class. Assuming all the data in all the experiments is completely random, what is the probability that at least one student will nonetheless find a "statistically significant effect" (p < 0.05) that can then be written up and get published.
rough answer: 99.4%
p < 0.05 seems to be chosen as that's a barrier that is not too harsh on the individual researcher (at very high significance levels it would be hard for anyone to run a powerful enough study in order to get published and keep their research job), but p-values are essentially defeated as a filter for genuine effects by the sheer number of experiments being run around the world.
In some senses, the answer here is simply to stop brandishing science as a social cudgel ("X is science; how dare you not believe it" or more often "how dare you employ/fund someone who does not believe it, they must be sacked") which puts it on a pedestal of being canonical truth all the time.
It is only the fact that science has recently been used more and more as a political stick to beat opponents with (and no, not just in climate and evolution, but right down to things like the best way to teach reading, or whether schools should be regulated or independent) that has meant that people think "science is broken" whenever it turns out to have got a wrong result. No, it's just expected to get it wrong fairly often. And in practice the most common "self-correction" process in science isn't a repeated experiment and published retraction, but just academics reading the paper, thinking "this one's garbage", and not basing their research from it.
Publication is not strictly a test of truth -- just a test of methodology and analysis. The test of truth always occurs in the mind of the reader.
That's incorrect. If you perform the exact same experiment twice, the chances are exactly 50% that the first result is more extreme than the second one (neglecting equal outcomes).
This illustrates nicely how hard it is to properly explain the p-value. Things would be more intuitive if people would report confidence intervals. I prefer "x is with a likelihood of 95% between a and b" a thousand times over "we measured x=c and reject the null hypothesis with 95% probability".
The chance is 50% if you condition only on the information you have before doing either experiment. But once you've done the first experiment, the chance of a more extreme result given what happened the first time may be much more or much less.
(Extreme example: Your experiment consists of rolling ten ordinary 6-sided dice. The null hypothesis is that they're fair dice, fairly rolled, in which case you expect a total not too different from 35 pips. All the dice come up 6. It is not now true that if you run the experiment again, you're as likely to get a more extreme result as you are to get a less extreme one!)
> confidence intervals [...] "x is with a likelihood of 95% between a and b"
But that isn't what a confidence interval means! A 95% confidence interval [a,b] means "If we ran the experiment lots of times, using the same method of computing the interval [a,b] each time, then in 95% of runs (in the long run) the true value would be in the interval [a,b] obtained on that run".
(What you described is what Bayesians call a "credible interval". Of course that interval depends on your prior.)
So 95% of the runs produce an interval that includes the true value. Doesn't that imply that when doing one run, there is a 95% chance of the resulting interval including the true value? I.e. the probability is 95% that the true value is in the interval?
The situation is similar to the one we discussed earlier where you do the same experiment twice. Before you do the experiment, your estimate of Pr(true value is in confidence interval) is 95%. But after you do the experiment, you have extra information: you know how the experiment turned out. And this may lead you to a different estimate of Pr(true value is in the confidence interval).
The equal probability you are describing is for the p-value itself being more extreme, not the result.
A fairly standard definition of the p-value (from wikipedia) says: "In statistics, the p-value is a function of the observed sample results (a statistic) that is used for testing a statistical hypothesis. Before the test is performed, a threshold value is chosen, called the significance level of the test, traditionally 5% or 1% and denoted as α."
What this description is missing though is the crucial importance of the fact that the threshold value is chosen before doing the analysis. And moreover, that the entire analysis plan has been chosen before doing the analysis. Because what the p-value is really telling you is the probability that on repeating the experiment (and its accompanying analysis!) you would see a result as or more extreme than what you observed.
If your experiment comprises "try all of the combinations of variables to see what gives me the best answer", the p-value you compute would need to be some very fancy test that took that into account... and you would see your analysis as having much less statistical power.
For a simple example, look at a statistically rigorous method for dealing with multiple hypothesis testing when you plan it in advance: https://en.wikipedia.org/wiki/Bonferroni_correction.
Of course p-hacking is bad. The problem isn't frequentist statistics or p-values though, its scientists not understanding the statistics that they use. If you want to use a p-value to help make a decision about a hypothesis, you have to commit to your analysis plan in advance.
edit: Furthermore, p-values were designed to deal with experimental data. If you're doing an observational study, perhaps you should use statistical tools designed for that purpose.
To sum up: when you have people who have no idea what they're doing do statistics, they will do it badly.