Re:A/B Testing Rigorously (without losing your job)
by gburt on Thursday January 24, 2013 @10:16PM
(#5113751)
I am strongly of the opinion that even if you got a non-significant result doing your A/B test, assuming we're testing something of no cost to change (like a web design that has already been deployed for the sake of the test), your point estimate for the effect is the "best data you have" and you should act on it.
To provide an example, you run a test and find Page 1 outperformed Page 2 by a factor of 1.1, but that was non-significant at your desired significance level (say, p=0.05 and power = 0.8). You should deploy page 1 instead of page 2, assuming there are no other costs associated with deploying page 1 instead of page 2, because your BEST GUESS RIGHT NOW is that page 1 is 1.1x better than page 2.
Re:A/B Testing Rigorously (without losing your job)
by rieter on Tuesday January 29, 2013 @12:24PM
(#5135115)
Yes, it is a guess that might be wrong, but it is a best guess you have. Staying on the old page is also a guess, which might be wrong, possibly with a larger likelihood.
To provide an example, you run a test and find Page 1 outperformed Page 2 by a factor of 1.1, but that was non-significant at your desired significance level (say, p=0.05 and power = 0.8). You should deploy page 1 instead of page 2, assuming there are no other costs associated with deploying page 1 instead of page 2, because your BEST GUESS RIGHT NOW is that page 1 is 1.1x better than page 2.