This is a more mathematically precise restatement of kerkeslager's explanation below and the obligatory xkcd [1]. The problem boils down to the fact that null results are not usually published, and that intuitive skepticism is not quantifiable.

Essentially, we want to get p(H0|x), but we need Bayes Law to get this from p(x|H0). But we need some notion of what priors to use. This is of course impossible to actually get, but if we published null studies then it would allow us to estimate it with better confidence. Alternatively we can ballpark it by saying how unexpected the result is, and how long the effect being explained has been studied.

The "common-sense" version of this is intuitive, that "extraordinary claims require extraordinary evidence". The more unexpected a result is, the lower the p-value has to be to be convincing.

The epidemiological version of this shows up in things like the Bradford Hill criteria [2], which include significance of association but also attempt to bring in plausibility.

[1] https://xkcd.com/882/

[2] https://en.wikipedia.org/wiki/Bradford_Hill_criteria

I'd argue that p(H0|x) is also (in most cases) pretty uninteresting from a scientific perspective, and so the whole "publish all the p-values" solution I think is only fighting half the battle. Gelman does a much better job of arguing this than I ever could [1], but this idea that rejecting the null is the "desired" outcome is the real problem. To put it another way, a low p-value is another way of saying "my model of the data generating process is bad", which is in most cases a pretty unsatisfactory outcome. As to the idea that priors are "impossible" to get, to quote Gelman [2] again, why "strain at the gnat that is the prior and swallow the ungainly camel that is the iid likelihood?

[1] https://statmodeling.stat.columbia.edu/2015/03/02/what-hypot... [2] https://statmodeling.stat.columbia.edu/2015/07/03/why-should...

Do you have a better of wording for inclusion in paper abstracts / article summaries than what I said? I'd love to hear one, but p(x|H0) is just as bad as p = 0.05.

The problem isn't that the wording is confusing, it's that p(x|H0) isn't a very useful thing to compute.

Everybody wants to compute p(H|x), the probility of a scientific hypothesis given the data. People want to do this so badly that they can't help interpreting the p-value that way.

You can actually compute p(H|x) if you use Bayesian stats.

With some caveats: the value you get for P(H|x) might be very sensitive to the priors you choose, and most people are not thoughtful enough about this.

Yes that is true, but there is a sense in which that just makes explicit a problem that already exists.

> You can actually compute p(H|x) if you use Bayesian stats.

You can't "compute" it for any useful meaning of the word "compute". You can estimate it intuitively, or you can try to look at how many pre-registered unpublished studies or null results have been published. Otherwise there's no way to even consider getting a grasp of what P(H) would be.

> Otherwise there's no way to even consider getting a grasp of what P(H) would be.

That's called picking a prior. Usually, pick the prior that maximizes the entropy on the given interval (zero prior knowledge).

For those reading and confused. Computing: p(H|x) = p(x|H)p(H)/p(x)

requires p(H) which the parent is suggesting is impossible to grasp.

If we want to know the probability that a coin is biased, we can assign probabilities to each hypothesis. For some people, p(bias=1/2)=1 (Dirac distribution), others might argue that p(bias=x)=1 for x in [0,1] (uniform distribution). Others might argue its some Beta function centered around 1/2. I believe what the parent is suggesting is that choosing which original belief we have in the system is a matter of philosophy, not computation.

Wait wait wait: you need the priors, also. Where do they come from?

That's definitely the hard part of switching to Bayesian methods. But the problem doesn't go away if you use p values, it just gets ignored.

Typically people doing research have prior information about what they are researching. E.g. previous studies have found effect sizes to be in some interval.

You can fallback on an uninformative prior or other tricks if you really want to model the idea that you know literally nothing about what you're researching. But that should be very rare.

You just make priors up, just like the 0.05 limit is made up. Making up a prior is basically asking the question: How likely does it seem that X has Y effect? Without having any data specific to that question.

Our 0.05 p-value limit is effectively just answering that question with a fixed 5%, no matter how ridiculous the proposition is.

"Our research creates a lot of confidence that the hypothesis is true. If our hypothesis happens to be false, then only 1 in 20 independent repetitions of our research (P = 0.05) will obtain the good results that we present here."