News for hackers. Stuff that matters.
Re:P-Values are not Error Probabilities (2003) [pdf] by conistonwater on Monday April 06, 2015 @08:04PM (#9331359)
> are irrationally attached to P values

This is actually a terrible problem. Most people, I think, do this because of nervous cluelessness: they are sure there is some theory that says what it means, but they aren't sure they understand it, so they insist on following some strict rules, whether they make sense or not.

I think the idea is that if you follow a strict rule given in a textbook (even if wrong or inapplicable), that makes people feel safer than if they try to figure the rule out for themselves. At least if they follow a rule, they aren't personally responsible for the outcome.

It can sometimes be quite difficult to explain why a certain rule should not be applied, because the same reasons for why they didn't understand the rule in the first place also make it difficult to understand explanations given by someone else.

Re:P-Values are not Error Probabilities (2003) [pdf] by wodenokoto on Monday April 06, 2015 @10:40PM (#9331739)
> so they insist on following some strict rules, whether they make sense or not.

I took my bachelor in business and we were always told to make rational business decision. That is, informed decisions, based on data.

As for actual statistics, we were told 2 rules of thumbs: You need at least 30 people in a group before you can make meaningful statistics and it is significant if P < 0.005

Re:P-Values are not Error Probabilities (2003) [pdf] by RA_Fisher on Monday April 06, 2015 @11:56PM (#9331929)
Unfortunately that advice is hugely misleading. The reason is that the p-value essentially tells you how repeatable your analysis will be, but doesn't say anything about how beneficial making the decision might be. Second, there is a relationship between how large the difference between outcomes are and how many people you need. So if taking your conversion rate from 10 to 15% would be really helpful to your business, you might need say thousands of people to notice that difference statistically.
Re:P-Values are not Error Probabilities (2003) [pdf] by cgearhart on Tuesday April 07, 2015 @01:55PM (#9335798)
> The reason is that the p-value essentially tells you how repeatable your analysis will be

Your statement contradicts the introductory paragraph of the article:

> "...the outcomes of these tests are mistakenly believed to yield the following information:... the probability that an initial finding will replicate; ..."

The p-value is the likelihood that the observed effect is due only to chance, not a measure of the repeatability of a result.

Re:P-Values are not Error Probabilities (2003) [pdf] by RA_Fisher on Tuesday April 07, 2015 @05:27PM (#9337280)
Ah yes, you're right! I made a statement about likelihood!
Re:P-Values are not Error Probabilities (2003) [pdf] by wodenokoto on Tuesday April 07, 2015 @10:36PM (#9338511)
This was meant as an example of how crazily statistics is taught and why people end up just following strict rules.