News for hackers. Stuff that matters.
Re:Appsumo reveals its A/B testing secret: only 1 out of 8 t... by bdonlan on Tuesday February 15, 2011 @07:54AM (#2221019)
The thing about non-statistically-significant tests is you can't, in fact, say "it is still more likely to make you money." You can say that the B set did in fact make you more money this time, but you can't rule out that being luck of the draw. It might even have a (very small) bias AGAINST making you money, but that bias got lost in the noise. That's why it's important to discard non-statistically-significant tests (or expand the sample size if you _really_ want to know...)

There are no comments on this comment yet.