Re:Don't calculate p-values without understanding them
by msellout on Friday May 10, 2013 @01:54AM
(#5684846)
No such thing as an always-valid p-value. Observations that were predictive yesterday may have no value tomorrow.
Re:Don't calculate p-values without understanding them
by btilly on Friday May 10, 2013 @06:12PM
(#5688963)
The usefulness of a statistical answer always, of course, depends on the validity of the statistical model used to generate said statistics. But that particular statistical model allows unlimited looks at the data, and unlimited stopping opportunities, without ruining the p-value.
That said, if you need to use p-values in A/B testing, you might want to read http://elem.com/~btilly/ab-testing-multiple-looks/part1-rigo... to get a procedure that gives always valid p-values, and then http://elem.com/~btilly/ab-testing-multiple-looks/part2-limi... for a more practical alternative and the caveats. (I still intend to return to the series, but not for a bit more.)