Re:3x Improvement: Great sign-up page split-test for a web-app
by 3pt14159 on Monday November 30, 2009 @01:21PM
(#968349)
Even if this has a G-test significance of 99.86%, it doesn't mean it is valid. You need to have AT LEAST 10 results for the formulas to work correctly. Also, saying 300% conversion is madness. I'm not even fully statistically convinced that the challenger is at ALL better, let alone a certainty of 300% better.
[ Parent
| on HN ]
Re:3x Improvement: Great sign-up page split-test for a web-app
by defen on Monday November 30, 2009 @04:42PM
(#968760)
If you model the two alternatives as Bernoulli processes with unknown success rates, and assume that the only difference between the two is what is specified on the page, and that they don't interact, and you assume a uniform prior on both parameters, B's conversion rate is higher than A's with probability 0.999572.