News for hackers. Stuff that matters.
Re:Most Winning A/B Test Results are Illusory [pdf] by gabemart on Sunday February 23, 2014 @06:06PM (#7288115)
I don't follow. Why would we expect 5% of those 90 cases to be false positives, and what relationship does the estimate of 5% have to p-value? I don't understand how p-value could ever be used to predict the number of false positives one would expect to observe in a bundle of arbitrary tests.
Re:Most Winning A/B Test Results are Illusory [pdf] by a_bonobo on Sunday February 23, 2014 @06:56PM (#7288308)
A p-value cutoff of 5% says that you have a 5% probability that you're wrong in rejecting the Null Hypothesis.

So if you test 100 times, you'd expect to wrongly reject the Null Hypothesis 5 times.

Re:Most Winning A/B Test Results are Illusory [pdf] by gabemart on Sunday February 23, 2014 @07:29PM (#7288426)
> A p-value cutoff of 5% says that you have a 5% probability that you're wrong in rejecting the Null Hypothesis.

I don't think this is right. A p-value cutoff of 0.05 doesn't, by itself, indicate anything about the underlying probability of incorrectly rejecting the null hypothesis. It tells you, in a test that meets your cutoff, if the null hypothesis is true, the chance of seeing a results as strong or stronger than the results observed in the test is 5% or less. But that can't tell you the chance you're wrong in rejecting the null hypothesis.

A 1% chance of seeing results as strong as your results if the null hypothesis is true does not mean that there's a 99% chance of the null hypothesis being false.

Regardless, even putting this disagreement to one side, I still don't see how the original author's point makes sense. He or she seems to be using the cutoff as an indication of the underlying false-positive probability for prospective tests, regardless of the results of those tests meet the cutoff or not.

Re:Most Winning A/B Test Results are Illusory [pdf] by dkural on Sunday February 23, 2014 @11:03PM (#7289024)
gabemart, you're right, a_bonobo, ronaldx, you guys are wrong. p-values are commonly misunderstood to mean that the result has "5% change of being wrong". That's not what a p-value is. Please go ahead and read the 'misunderstandings' section on p-values in wikipedia.
Re:Most Winning A/B Test Results are Illusory [pdf] by ronaldx on Monday February 24, 2014 @05:41AM (#7290086)
sigh, no. When the author says "p-value cutoff", this refers to the significance level. I interpreted this correctly.