News for hackers. Stuff that matters.
Re:Don't calculate p-values without understanding them by btilly on Thursday May 09, 2013 @11:00AM (#5680759)
It is great to understand this stuff. But people don't. And I am personally of the belief that p-values are popular exactly because they are so easy to misunderstand as the answer to the question that we want to ask (what is the probability that we are right).

That said, if you need to use p-values in A/B testing, you might want to read http://elem.com/~btilly/ab-testing-multiple-looks/part1-rigo... to get a procedure that gives always valid p-values, and then http://elem.com/~btilly/ab-testing-multiple-looks/part2-limi... for a more practical alternative and the caveats. (I still intend to return to the series, but not for a bit more.)

Re:Don't calculate p-values without understanding them by msellout on Friday May 10, 2013 @01:54AM (#5684846)
No such thing as an always-valid p-value. Observations that were predictive yesterday may have no value tomorrow.
Re:Don't calculate p-values without understanding them by btilly on Friday May 10, 2013 @06:12PM (#5688963)
The usefulness of a statistical answer always, of course, depends on the validity of the statistical model used to generate said statistics. But that particular statistical model allows unlimited looks at the data, and unlimited stopping opportunities, without ruining the p-value.