It’s not just p=0.048 vs. p=0.052

Re:It’s not just p=0.048 vs. p=0.052 by dash2 on Wednesday September 11, 2019 @08:47AM (#20939459)

They're right. If the null hypothesis is true, then the probability of getting any p value is equal. Put it another way, the p value is the probability of getting the observed data (or more extreme) under the null. So, under the null, 10% of the time you will get data with a p value of 10% or less; 20% of the time, you will get data with a p value of 20% or less; and so on. And that's the uniform distribution!

Here's an R example to play with:

  pvals <- replicate(10000, {
    x <- rnorm(100)
    y <- rnorm(100)
    t.test(x, y)$p.value
  })
  plot(density(pvals))

That will plot you a nice uniform line on [0, 1].

(NB: I have no idea why OP talked about p values following a normal distribution. That doesn't make sense to me, and I think the post has been deleted.)

[ Parent | on HN ]

Re:It’s not just p=0.048 vs. p=0.052 by mehrdadn on Wednesday September 11, 2019 @03:46PM (#20943908)

So I don't have (or know) R, but I do have access to Mathematica, and this is most definitely not giving me a uniform distribution (how could it?!):

  << HypothesisTesting`
  With[{n = 10000000, dist = NormalDistribution[]},
   Histogram[Last[NormalPValue[RandomVariate[dist, n] - RandomVariate[dist, n]]], 500]]

Why is your x variable though? If H0 is true shouldn't your x be fixed?

I think if you remove the subtraction though then you do get a uniform distribution -- in which case I see what the claim is, yeah. Wasn't really clear to me earlier but indeed, getting p = 5% means you have a 5% chance of getting observations that extreme, so I guess it is uniformly distributed!

[ Parent | on HN ]

Re:It’s not just p=0.048 vs. p=0.052 by dash2 on Friday September 20, 2019 @04:59AM (#21024580)

You'd get the same result whether you test the hypothesis y > x (with both random variables) or y > 0. Under the null, the p-value is uniform on [0, 1], whatever that null is. It follows from the definition of a p value.

(Disclaimer: I am not a real statistician....)

[ Parent | on HN ]