News for hackers. Stuff that matters.
Re:Reproducibility Initiative gets $1.3M grant to validate 5... by mjn on Wednesday October 16, 2013 @04:08PM (#6561848)
Not necessarily 5%, no. The p-value is controlling the false-positive rate, i.e. it constrains the likelihood that a seemingly-significant result was really just by chance. Therefore, if we were now given the real ground truth, it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all.

But you seem to be positing the reverse: a hypothetical case where we assume that the results of the original studies were in fact correct (i.e. ground truth), and we want a sort of false-negative rate, the chance that a fact would not be confirmed by a replication study, despite actually being true. That depends on several things about the replication attempt, such as sample size and methodology, summarized into the concept of statistical power: http://en.wikipedia.org/wiki/Statistical_power

The overall failure-to-replicate rate would involve both aspects, with four possible outcomes at varying likelihood: 1) a correct result that was confirmed; 2) a correct result that failed to be confirmed; 3) an incorrect result that was confirmed anyway; and 4) an incorrect result that failed to be confirmed. Obviously it would be ideal if #1 and #4 were much higher than #2 and #3.

Re:Reproducibility Initiative gets $1.3M grant to validate 5... by gwern on Wednesday October 16, 2013 @06:25PM (#6562480)
> That depends on several things about the replication attempt, such as sample size and methodology, summarized into the concept of statistical power: http://en.wikipedia.org/wiki/Statistical_power

It's not just power, you also need prior odds if you want to make an unconditional estimate. This is pretty much the core of Ioannides's famous paper http://www.plosmedicine.org/article/info:doi/10.1371/journal... - even after taking power into account, the replication rate could be anywhere from 0 to say 80% depending on the prior odds.

Re:Reproducibility Initiative gets $1.3M grant to validate 5... by gabemart on Wednesday October 16, 2013 @04:30PM (#6561982)
Thanks, very interesting. It was silly of me to assume that the p-value could be used in that way when, as you explain, it's not measuring or indicating that at all.
Re:Reproducibility Initiative gets $1.3M grant to validate 5... by gwern on Wednesday October 16, 2013 @06:27PM (#6562486)
Don't feel too bad. It's a very common mistake; see some of the citations in https://en.wikipedia.org/wiki/P-value#Misunderstandings and http://lesswrong.com/lw/g13/against_nhst/
Re:Reproducibility Initiative gets $1.3M grant to validate 5... by yummyfajitas on Wednesday October 16, 2013 @06:05PM (#6562401)
it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all.

Not quite. Assume P of the tests are truly positive, and N are truly negative.

We expect to see approximately 0.05 x N + A x P positives in the entire sample, where A is the probability of a false negative. So the fraction of true positives not replicated is likely to be 0.05 x N / (0.05 x N + A x P).