Re:Appsumo reveals its A/B testing secret: only 1 out of 8 t...
by tomjen3 on Tuesday February 15, 2011 @06:41AM
(#2220879)
I wouldn't call them failed tests. In science a null result mean that you didn't learn anything but in this case who cares if p is 0.6? It is still more likely to make you money.
[ Parent
| on HN ]
Re:Appsumo reveals its A/B testing secret: only 1 out of 8 t...
by bdonlan on Tuesday February 15, 2011 @07:54AM
(#2221019)
The thing about non-statistically-significant tests is you can't, in fact, say "it is still more likely to make you money." You can say that the B set did in fact make you more money this time, but you can't rule out that being luck of the draw. It might even have a (very small) bias AGAINST making you money, but that bias got lost in the noise. That's why it's important to discard non-statistically-significant tests (or expand the sample size if you _really_ want to know...)
Re:Appsumo reveals its A/B testing secret: only 1 out of 8 t...
by barrkel on Tuesday February 15, 2011 @07:40AM
(#2220983)
I think precisely the point of not being able to reject the null hypothesis is that you can't be sure it's more likely to make you more money; otherwise you could have required a lower confidence level and still rejected the null hypothesis.