Re:Most Winning A/B Test Results are Illusory [pdf]
by iambateman on Monday February 24, 2014 @09:12AM
(#7290796)
So if my site has 10k hits/month and I don't have a year, am I left to "stick with my gut" and qualitative user tests?
[ Parent
| on HN ]
Re:Most Winning A/B Test Results are Illusory [pdf]
by gk1 on Monday February 24, 2014 @09:32AM
(#7290927)
You can still run successful tests. Here's a handy calculator you can use to approximate a required sample size for your test: http://www.evanmiller.org/ab-testing/sample-size.html
Re:Most Winning A/B Test Results are Illusory [pdf]
by ernopp on Monday February 24, 2014 @09:30AM
(#7290909)
Well it depends on your conversion rate. But assuming it's ~few percent, then yes it will be hard to measure anything other than very very large effects in conversion rate unless you're willing to wait quite a long time.
You can measure more upper-funnel things though like I said in the GP, which can be very helpful, especially in combination with qualitative feedback, although this depends on what exactly your business is...