Re:3x Improvement: Great sign-up page split-test for a web-app
by mcantor on Monday November 30, 2009 @12:53PM
(#968284)
I'm extremely rusty on my statistics, and I upvoted this because I find A/B tests interesting, but... are these numbers statistically significant? For the population of internet users, are they actually practically significant? It just seems like the sample sizes and differences aren't really big enough to draw solid conclusions from. It doesn't say how long each test lasted for, either--what if the second test was done during peak hours?
[ Parent
| on HN ]
Re:3x Improvement: Great sign-up page split-test for a web-app
by TimothyFitz on Monday November 30, 2009 @01:04PM
(#968306)
This is exactly why I don't read other people's A/B experiment results. I haven't seen a single A/B experiment that listed it's statistical significance (and how that was calculated). I fear that bad data is worse than no data at all.
Re:3x Improvement: Great sign-up page split-test for a web-app
by brown9-2 on Monday November 30, 2009 @03:04PM
(#968551)
Google Website Optimizer will give you a plus minus rate for your estimated conversion rate, which helps you to at least estimate what the significance/confidence is.
For example, one of the pages in one of my current tests shows up as "Est. Conversion Rate: 17.0% +/- 1.4%".