News for hackers. Stuff that matters.
Re:Not Even Scientists Can Easily Explain P-values by nonbel on Wednesday November 25, 2015 @06:13PM (#10630402)
>"You haven't shown anything where a true null hypothesis uniformly generates p-values between 0 and 1."

Try this in R. It calculates p-values for testing the hypothesis that two groups (n=10) come from the same distribution (normal with mean=0, sd=1). It does 100,000 such comparisons, then makes a histogram. This results in approximately a uniform(0,1) distribution of p-values:

hist(replicate(10^5,t.test(rnorm(10),rnorm(10),var.equal=T)$p.value))

If you increase to 10^6, 10^7, etc it will converge upon the uniform distribution. That is what people mean.

Edit:

Or something like this. Keep adding to the sample, the p-value will random walk between 0 and 1. The mean will converge onto 0.5:

n=10; Nsim=10000

a=rnorm(n); b=rnorm(n)

p=matrix(nrow=Nsim,ncol=2)

for(i in 1:Nsim){

p[i,1]=length(a)

p[i,2]=t.test(a,b,var.equal=T)$p.value

a=c(a,rnorm(n)); b=c(b,rnorm(n))

plot(p[,1],p[,2],ylim=c(0,1), ylab="pvalue", xlab="sample size")

}

There are no comments on this comment yet.