Re:P-Values are not Error Probabilities (2003) [pdf]
by RA_Fisher on Monday April 06, 2015 @11:56PM
(#9331929)
Unfortunately that advice is hugely misleading. The reason is that the p-value essentially tells you how repeatable your analysis will be, but doesn't say anything about how beneficial making the decision might be. Second, there is a relationship between how large the difference between outcomes are and how many people you need. So if taking your conversion rate from 10 to 15% would be really helpful to your business, you might need say thousands of people to notice that difference statistically.
Re:P-Values are not Error Probabilities (2003) [pdf]
by cgearhart on Tuesday April 07, 2015 @01:55PM
(#9335798)
> The reason is that the p-value essentially tells you how repeatable your analysis will be
Your statement contradicts the introductory paragraph of the article:
> "...the outcomes of these tests are mistakenly believed to yield the following information:... the probability that an initial finding will replicate; ..."
The p-value is the likelihood that the observed effect is due only to chance, not a measure of the repeatability of a result.
Re:P-Values are not Error Probabilities (2003) [pdf]
by RA_Fisher on Tuesday April 07, 2015 @05:27PM
(#9337280)
Ah yes, you're right! I made a statement about likelihood!
Re:P-Values are not Error Probabilities (2003) [pdf]
by wodenokoto on Tuesday April 07, 2015 @10:36PM
(#9338511)
This was meant as an example of how crazily statistics is taught and why people end up just following strict rules.
I took my bachelor in business and we were always told to make rational business decision. That is, informed decisions, based on data.
As for actual statistics, we were told 2 rules of thumbs: You need at least 30 people in a group before you can make meaningful statistics and it is significant if P < 0.005