Re:P-Values are not Error Probabilities (2003) [pdf]
by RA_Fisher on Monday April 06, 2015 @11:56PM
(#9331929)
Unfortunately that advice is hugely misleading. The reason is that the p-value essentially tells you how repeatable your analysis will be, but doesn't say anything about how beneficial making the decision might be. Second, there is a relationship between how large the difference between outcomes are and how many people you need. So if taking your conversion rate from 10 to 15% would be really helpful to your business, you might need say thousands of people to notice that difference statistically.
[ Parent
| on HN ]
Re:P-Values are not Error Probabilities (2003) [pdf]
by cgearhart on Tuesday April 07, 2015 @01:55PM
(#9335798)
> The reason is that the p-value essentially tells you how repeatable your analysis will be
Your statement contradicts the introductory paragraph of the article:
> "...the outcomes of these tests are mistakenly believed to yield the following information:... the probability that an initial finding will replicate; ..."
The p-value is the likelihood that the observed effect is due only to chance, not a measure of the repeatability of a result.