News for hackers. Stuff that matters.
Re:P-Values are not Error Probabilities (2003) [pdf] by capnrefsmmat on Monday April 06, 2015 @10:30PM (#9331704)
The difference between p values and alpha levels is a bit subtle, and when I first read this paper (while preparing my book, Statistics Done Wrong) it took me a while to figure out.

Here's the idea. If you set alpha = 0.05, you will declare statistically significant any result that gets a p value of 0.05 or less. When there is no statistically significant difference to be found, you will have a 5% chance of falsely detecting one.

But crucially, this applies on average to all tests you conduct with this alpha level. Even if an individual test gets p = 0.000001 or p = 0.04, the overall false positive rate will be 5%.

More succinctly, it doesn't make sense to ask for the false positive rate of a single test. What does that even mean? You can only ask for the false positive rate of a procedure you use many times. So you can't get p = 0.01 and declare this means you have a false positive rate of 1%.

Re:P-Values are not Error Probabilities (2003) [pdf] by philh on Tuesday April 07, 2015 @07:31AM (#9333188)
Possibly worth clarifying: the false positive rate (https://en.wikipedia.org/wiki/False_positive_rate) is "probability that a test will return positive, conditional on the hypothesis being false". It's the rate of false positives within the set of negatives, not the rate of false positives within all tests.