Its a meta observation of how we view p-values. He's saying that the same logic behind p-values and declaring something "statistically significant" when p < X (often 0.05), falls apart when applied to differences between two observed p-values. Essentially, you would say there is no "statistically significant" difference between an experiment that resulted in a p-value of 0.2 and one that resulted in a p-value of 0.05.

And the argument will hold no matter what threshold you choose for rejecting the null hypothesis. You can choose to reject if p > X, and for any X, there will be values greater than X that, applying this meta-logic, are not statistically different from X.

> At what point would this author say something is not consistent with the null hypothesis?

Gelman's argument, I presume, is against the idea of significance testing as a whole. Declaring something "statistically significant" is in itself a very problematic thing, as it distills the entire phenomenon, the uncertainty surrounding the experiment, and the uncertainty surrounding the researcher's decisions to a single, binary conclusion.

Gelman is a Bayesian (perhaps the most famous modern Bayesian), and the Bayesian philosophy is to focus on producing a posterior distribution of the phenomenon being studied. I presume the alternative to significance and null hypothesis testing that he was suggest would be something closer to a model where people are reporting their priors/data/posteriors, and the discussion focuses around the implications and replication of those.

Nasim Taleb wrote extensively on this.

> P-values are shown to be extremely skewed and volatile, regardless of the sample size n, and vary greatly across repetitions of exactly same protocols under identical stochastic copies of the phenomenon; such volatility makes the minimum p value diverge significantly from the "true" one. Setting the power is shown to offer little remedy unless sample size is increased markedly or the p-value is lowered by at least one order of magnitude.

https://arxiv.org/abs/1603.07532

Video: https://www.youtube.com/watch?v=8qrfSh07rT0

Are you sure this answers my question? Note that I was neither asking why p-hacking is bad nor even why p-values are bad.

Yes, it answers your question.

What you call "p-value" is a sample from the "p-value distribution" of your experiment.

Taleb shows you can sample a p-value of 0.05 when the actual "true" p-value is 0.12.

I guess I have to sleep on this because at a quick glance I can't really make sense of how it answers my question.

I do find it ironic though that this is so difficult to explain that I apparently have to read a paper to understand it... I would've thought the blog post was trying to explain things in simple terms...

Well, if this stuff was easy to understand, we wouldn't be having a crisis because of it, would we?

Probability is highly non-intuitive (kind of like quantum mechanics), so most people (including scientists) don't understand it and just memorize "protocols" and "formulas" and "p-value stuff".

https://www.tandfonline.com/doi/full/10.1080/00031305.2018.1...

The crisis isn't caused by ignorance. It isn't like if people took some courses on stats or just chose another system we'd resolve everything. Replace p values with any other system for distinguishing publishable work and non publishable work and you end up in the same place.

Science is messy but eventually consistent. Everybody should be cautious of single results and instead rely on the community to synthesize those results into predictive models.

How about evaluating a scientist beyond how many "publishable" discoveries he makes? There are so many other roles a scientist can play in the community beyond that.

We've learned not to value developers based on lines of code written, and to value refactoring and elimination of code, scientist can also do the equivalent. Scrutinize, weed out, reinterpret old things, mentor others, etc. Much stuff beyond finding p<0.05.

But such things are rather seen a side quests, secondary purposes, almost like a hobby (e.g. writing a textbook).

You are not answering his question, which is about the situation where the null hypothesis is true (what Taleb would call a "true" p-value of 0.5; for some reason he decided to define as the "true" p-value the average of the distribution - but it's worth noticing that there is not such a thing as a "true" p-value).

The only thing that matters to answer his question is the sampling distribution of the p-value when the null hypothesis is true, which is uniform in [0 1].

> Could someone clarify what is meant here?

If the null hypothesis is true every p-value is equally likely, by construction. However, getting a extreme value (under a small threshold) is less likely that getting not-so-extreme value (under a not-so-small threshold). The probability of getting a p-value below 0.05 is 5%, the probability of getting a p-value below 0.25 is 25%, the probability of getting a p-value below 1 is 100%.

If you show me a drug that cured 2 out of 10 ebola patients, when it's known that 10% of patients recover without treatment, I won't be impressed (high p-value). If you show me a drug that cured 9 out of 10 patients you're onto something (low p-value).

> At what point would this author say something is not consistent with the null hypothesis?

The author just doesn't like hypothesis testing. His view is that the null hypothesis is always false and everything is consistent with it.

> The probability of getting a p-value below 0.05 is 5%, the probability of getting a p-value below 0.25 is 25%, the probability of getting a p-value below 1 is 100%.

But getting a p value between 0.82 and 0.87 is also just 5% probability, so could also be seen as a probably-won't-happen-by-chance event. Sure it's clear most of the time which 5% is the significant one,but not always, for example paradoxically the same result can be significant or not, depending on what your intended stopping criterion was, even if you happened to have to stop at the same time using either criterion. This is because the two variants would lead to declaring a different 5% portion of the null's possible outcomes as the significant part.

He is coming at that conclusion from a Bayesian point of view to statistics. He is seeing the p-value as a random variable that can take values from 0 to 1 and follows some distribution. Under these hypotheses, observing a p-value of 0.20 and 0.005 is completely reasonable even if unlikely. Those are just two draws from a random variable.

Edit. Under Bayesian statistics testing the null hypothesis is a moot point as it becomes possible to directly model the distribution of the possible effects. Thinking of it as being able to look at a picture of something (the p-value) vs looking at a movie of it (the distribution of the effects).

That's not even a bayesian point of view, even in a frequentist setting the p-value is a conditional probability, conditioned on the dataset.

What he says is (I gather) worse: those events are only separated be 1.1std deviations, which is little.

I think that's what he's saying too, but what is that supposed to show? Is he arguing against some claim that every interval of 1 standard deviation is equally significant? Did anybody make this claim? So far as I know, nobody considers (say) a 6-sigma effect to be 6 times stronger than a 1-sigma effect...

I don’t get it either. It’s a trivial consequence of having a threshold: if we say two cities are “far” when they are at least 1000 miles away then Washington D.C. is not far from Jacksonville while Boston is far from Jacksonville, even though Boston is not far from Washington.

That's not quite the problem.

Let's assume you've already decided in advance what "far" means.

Without moving either city from its current location, the same experiment can give you "very far" and "very close" in identical replications.

I’m commenting on pfortuny’s (correct) interpretation “those events are only separated be 1.1std deviations, which is little” of the original post by Gelman (the difference between a significant result and a non-significant result may not be significant = a city which is far from Jacksonville and a city which is not far from Jacksonville may not be far from each other).

I still don't get it. What p-value observations would not be "reasonable" here? It seems to me he's saying that anything between 0 and 1 is completely reasonable, which is a completely pointless statement as I see it.

His point is that the whole point of the field of statistics is that cherry picking results or getting lucky once is not proof of anything, unlike nonstatistical mathematics where one example is enough to justify a claim.

> He is seeing the p-value as a random variable that can take values from 0 to 1 and follows some distribution.

That is the frequentist point of view!

I have no idea.

For a identical normal populations, repeating an experiment produces p<=0.2 20% of the time, and produces p<=0.005 0.5% of the time.

A coin comes up on the same side 3 times in a row vs 8 times in a row...I have no idea why we should shrug and consider the plausibility of coin bias in these two cases about the same.

---

EDIT: I he means that in the case of an actual difference in the populations, p=0.2 and p=0.005 are both pretty likely outcomes.

When the populations are the same, p=0.2 and p=0.005 are quite different happenings.

This is because p-value methods doesn't worry very much about type II errors.

The distribution of the p-value, given that the null is true, is uniform between 0 and 1.

IF the null is true, you're equally likely to get a p-value of 0.01 and 0.87.

Say you run an experiment. You get some data. From that data you calculate a p-value. The data has randomness, so the p-value is a random variable and has a distribution. By chance you can get results that look significant. By chance you can get results that look insignificant.

Well, if there is no effect (the effect-size is zero), two different experimenters will likely see two different p-values regardless of how large an experiment either of them runs.

So by this logic "it is completely consistent with the null hypothesis to see p-values of 0.00000001 and 0.99999999 from two replications of the same damn experiment"?

At which point, what is even the point of this statement?

My understanding (Someone please correct me if I'm wrong) is that while the values can range from limx->0 and limx->1 , the actual distribution will be skewed in cases where it is statistically significant. E.g. If you repeat the experiment 100x you could find a p=.999999 but more likely they'll be close to .05 (or whatever alpha you choose)

If the null hypothesis is true the p-value is distributed uniformly in [0 1] (at least ignoring discrete data, composite hypothesis and other cases which are not easy).

The distribution will be different when then true hypothesis is not true (but you may also get non-significant results even if the null hypothesis is not true).

I’m not sure if that’s what you mean by “the actual distribution will be skewed in cases where it is statistically significant.”

Sorry I was definitely kinda vague - I meant when the alternative hypothesis is true the distribution should be skewed towards 0.

The point is that you need to replicate experiments and not cherry pick the one with the highest p-value.

.

> It means that if the null hypothesis were true, you expect your p-values to be a random variable contained inside a nice bell shaped normal. 0.005 is dead center so it's very likely but O.2 which seems very unprobable is actually only 1std further, it's well inside the bell curve.

I'm completely lost here. How is 0.005 "dead center"? Are you assuming p = 0 is the center? Are there negative p-values I'm not seeing that somehow balance the positive ones?

How can a random variable that's strictly between 0 and 1 even follow a bell curve?

Check out these links [0] [1] or google for "p value distribution" or "p curve"

[0] http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/HtestPValue/t...

[1] https://en.wikipedia.org/wiki/P-value#Distribution

I'm not following those links either. How is p uniformly distributed under H0? If you assume H0 then obtaining a p-value near 0 is going to be damn impossible. Whereas obtaining one similarly close to 0.5 is going to be ridiculously more likely.

Am I severely lacking sleep and going crazy or something? Maybe I should check back in like half a day to see what people have said, I feel like I must be completely confused right now because literally nothing I've read so far makes sense to me.

They're right. If the null hypothesis is true, then the probability of getting any p value is equal. Put it another way, the p value is the probability of getting the observed data (or more extreme) under the null. So, under the null, 10% of the time you will get data with a p value of 10% or less; 20% of the time, you will get data with a p value of 20% or less; and so on. And that's the uniform distribution!

Here's an R example to play with:

  pvals <- replicate(10000, {
    x <- rnorm(100)
    y <- rnorm(100)
    t.test(x, y)$p.value
  })
  plot(density(pvals))

That will plot you a nice uniform line on [0, 1].

(NB: I have no idea why OP talked about p values following a normal distribution. That doesn't make sense to me, and I think the post has been deleted.)

So I don't have (or know) R, but I do have access to Mathematica, and this is most definitely not giving me a uniform distribution (how could it?!):

  << HypothesisTesting`
  With[{n = 10000000, dist = NormalDistribution[]},
   Histogram[Last[NormalPValue[RandomVariate[dist, n] - RandomVariate[dist, n]]], 500]]

Why is your x variable though? If H0 is true shouldn't your x be fixed?

I think if you remove the subtraction though then you do get a uniform distribution -- in which case I see what the claim is, yeah. Wasn't really clear to me earlier but indeed, getting p = 5% means you have a 5% chance of getting observations that extreme, so I guess it is uniformly distributed!

You'd get the same result whether you test the hypothesis y > x (with both random variables) or y > 0. Under the null, the p-value is uniform on [0, 1], whatever that null is. It follows from the definition of a p value.

(Disclaimer: I am not a real statistician....)

> How is p uniformly distributed under H0?

That's the very definition of a p-value! The mapping of data to p-values is chosen to have a uniform distribution of p-values when the data is distributed according to the null hypothesis. That's the property that makes p-values interesting.

At least for continuous statistics, the p-value is uniformly distributed when the null hypothesis is true.

I don’t think you understand what a p-value is. The p-value is a percentage output of testing whether a given normal distribution actually has a non-zero mean. It is phrased in terms of the null hypothesis. So a P < 0.05 means that there is less than a 5% chance that the null hypothesis is true. Conversely, this means that there is a 95% chance that what you’re testing is true.

> So a P < 0.05 means that there is less than a 5% chance that the null hypothesis is true. Conversely, this means that there is a 95% chance that what you’re testing is true.

See https://en.wikipedia.org/wiki/P-value#Basic%20concepts

You don't understand what a p-value is.

> a P < 0.05 means that there is less than a 5% chance that the null hypothesis is true.

In other words, P(H0 | X) where H0 is the null hypothesis being true and X is the data observed. But that is not what a p-value is, they actually represent P(X | H0).

mruts' (and most people who studied a little statistics) misunderstanding of the basic definition is exactly what author and others have been campaigning against for years.