News for hackers. Stuff that matters.
Re:P-Values are not Error Probabilities (2003) [pdf] by cl42 on Monday April 06, 2015 @06:19PM (#9331032)
Thanks for sharing this. In general, it's a real shame how few people know how to interpret and use P values correctly. We work with a lot of businesses who ask us to compare populations (e.g., through A/B tests) and people either (a) don't care about the significance between population differences, or (b) are irrationally attached to P values.

Case in point #1: debating whether a P value of 0.051 versus 0.049 is a major difference in the significance test.

Case in point #2: a P value of <0.001 but with extremely low differences in means between populations. With enough data, everything is significant!

End rant. :)

Re:P-Values are not Error Probabilities (2003) [pdf] by conistonwater on Monday April 06, 2015 @08:04PM (#9331359)
> are irrationally attached to P values

This is actually a terrible problem. Most people, I think, do this because of nervous cluelessness: they are sure there is some theory that says what it means, but they aren't sure they understand it, so they insist on following some strict rules, whether they make sense or not.

I think the idea is that if you follow a strict rule given in a textbook (even if wrong or inapplicable), that makes people feel safer than if they try to figure the rule out for themselves. At least if they follow a rule, they aren't personally responsible for the outcome.

It can sometimes be quite difficult to explain why a certain rule should not be applied, because the same reasons for why they didn't understand the rule in the first place also make it difficult to understand explanations given by someone else.

Re:P-Values are not Error Probabilities (2003) [pdf] by wodenokoto on Monday April 06, 2015 @10:40PM (#9331739)
> so they insist on following some strict rules, whether they make sense or not.

I took my bachelor in business and we were always told to make rational business decision. That is, informed decisions, based on data.

As for actual statistics, we were told 2 rules of thumbs: You need at least 30 people in a group before you can make meaningful statistics and it is significant if P < 0.005

Re:P-Values are not Error Probabilities (2003) [pdf] by RA_Fisher on Monday April 06, 2015 @11:56PM (#9331929)
Unfortunately that advice is hugely misleading. The reason is that the p-value essentially tells you how repeatable your analysis will be, but doesn't say anything about how beneficial making the decision might be. Second, there is a relationship between how large the difference between outcomes are and how many people you need. So if taking your conversion rate from 10 to 15% would be really helpful to your business, you might need say thousands of people to notice that difference statistically.
Re:P-Values are not Error Probabilities (2003) [pdf] by cgearhart on Tuesday April 07, 2015 @01:55PM (#9335798)
> The reason is that the p-value essentially tells you how repeatable your analysis will be

Your statement contradicts the introductory paragraph of the article:

> "...the outcomes of these tests are mistakenly believed to yield the following information:... the probability that an initial finding will replicate; ..."

The p-value is the likelihood that the observed effect is due only to chance, not a measure of the repeatability of a result.

Re:P-Values are not Error Probabilities (2003) [pdf] by RA_Fisher on Tuesday April 07, 2015 @05:27PM (#9337280)
Ah yes, you're right! I made a statement about likelihood!
Re:P-Values are not Error Probabilities (2003) [pdf] by wodenokoto on Tuesday April 07, 2015 @10:36PM (#9338511)
This was meant as an example of how crazily statistics is taught and why people end up just following strict rules.
Re:P-Values are not Error Probabilities (2003) [pdf] by return0 on Monday April 06, 2015 @06:23PM (#9331054)
Amen for #2! It's like the entire field of biology is an endless quest for low p-values, regardless if the hypothesis is even interesting. It's like people are not interested to think, they just want to publish something significantly differnt.
Re:P-Values are not Error Probabilities (2003) [pdf] by cl42 on Monday April 06, 2015 @06:47PM (#9331143)
haha, Sociology as well -- especially now that the web provides huge amounts of behavioral data.

I much prefer how machine learning folks tend to approach predictive accuracy, though I guess that's not quite the same as understanding relationships between specific variables while controlling for others.

Re:P-Values are not Error Probabilities (2003) [pdf] by sdenton4 on Monday April 06, 2015 @07:09PM (#9331217)
There is a notion of 'feature importance,' which especially comes up in decision trees and random forests, giving a notion of how much a particular feature contributes to the overall prediction. It seems like combining predictive power with feature importance would be an interesting alternate route to demonstrating important correlations. (For example, maybe a model predicts lung cancer with 90% precision, and 'is_smoker' has a 80% feature importance.) Of course, these importances depend a lot on the other features used by the model! If you include a lot of junk features and/or exclude other important features, the importance of your pet feature will shoot up.
Re:P-Values are not Error Probabilities (2003) [pdf] by cl42 on Monday April 06, 2015 @09:24PM (#9331533)
Hmm, interesting -- I never considered the idea of including junk features to bias model's preferences of whatever theoretically ambiguous idea you're trying to promote. That's actually brilliant.
Re:P-Values are not Error Probabilities (2003) [pdf] by mbq on Tuesday April 07, 2015 @06:53AM (#9333032)
Shameless plug; I'm a co-author of a method that leverages adding artificial junk features and removing original ones that are likely nonsense to approximate the set of all features that are relevant to the problem (rather than standard make best model, which may be pretty deceiving). https://m2.icm.edu.pl/boruta
Re:P-Values are not Error Probabilities (2003) [pdf] by shasta on Monday April 06, 2015 @06:56PM (#9331173)
Shoes reduce cholesterol.
Re:P-Values are not Error Probabilities (2003) [pdf] by cl42 on Monday April 06, 2015 @09:23PM (#9331529)
And aren't ice cream sales dependent on murder rates?
Re:P-Values are not Error Probabilities (2003) [pdf] by copperx on Wednesday April 08, 2015 @03:31AM (#9339205)
Did you just discredit Freakonomics?
Re:P-Values are not Error Probabilities (2003) [pdf] by Sharlin on Tuesday April 07, 2015 @08:43AM (#9333519)
> they just want to publish something

What's going on in the academia is a classical example of perverse incentives. If your career depends on churning out soul-crushingly bad or insignificant papers that fulfill some superficial criteria of "scienceness" then that's what you'll do. It's dangerously close to cargo cult science.

Re:P-Values are not Error Probabilities (2003) [pdf] by joshdick on Tuesday April 07, 2015 @12:04AM (#9331947)
Addressing that first point is "The Difference Between 'Significant' and 'Not Significant' is not Itself Statistically Significant" by Andrew Gelman and Hal Stern.

PDF link: http://www.stat.columbia.edu/~gelman/research/published/sign...

Re:P-Values are not Error Probabilities (2003) [pdf] by stdbrouw on Tuesday April 07, 2015 @05:53AM (#9332839)
Nitpick: while that paper does mention the basic idea that there's no significant difference between 0.051 and 0.049, it's more about another common mistake: testing the difference between two groups by performing two separate statistical tests and then comparing the outcomes (a <> b), rather than the correct approach which is to do a single test for the difference between groups (a - b <> 0).
Re:P-Values are not Error Probabilities (2003) [pdf] by Scoundreller on Monday April 06, 2015 @08:12PM (#9331372)
One of my favourite go-to phrases in health care is: "Statistically significant, but a clinically insignificant difference."
Re:P-Values are not Error Probabilities (2003) [pdf] by 6502nerdface on Tuesday April 07, 2015 @12:57AM (#9332096)
In quant finance, my colleagues and I say, "OK, it's statistically significant, but is it economically significant?" I love that we have parallel sayings in totally different fields.
Re:P-Values are not Error Probabilities (2003) [pdf] by cl42 on Monday April 06, 2015 @09:21PM (#9331525)
I love it! That's actually a really good way of looking at it, right? Unfortunately for researchers, I'd argue that "Statistically significant, but a clinically insignificant difference." is equivalent to "Theoretically useless, and technically irrelevant." :)
Re:P-Values are not Error Probabilities (2003) [pdf] by remarkEon on Tuesday April 07, 2015 @02:18PM (#9335946)
Right, you're describing statistical significance as it relates to practical significance.