Re:A/B Testing Rigorously (without losing your job)
by btilly on Thursday January 24, 2013 @07:18PM
(#5113162)
In this article I was going for the highest possible standard. The ins and outs of one-sided versus two-sided was not a topic that I wanted to explore.
In the next article I plan to relax things a lot farther than just one-sided versus two-sided tests!
Re:A/B Testing Rigorously (without losing your job)
by majormajor on Thursday January 24, 2013 @07:25PM
(#5113187)
Gotcha, thanks. And one- vs two-sided alone would hardly be a huge win, I was just curious since I didn't remember seeing it brought up at all in any of the prior articles I'd read.
It should let you get away with a lower sample size at the expense of possibly only being able to conclude "A's better" or "A's not better" instead of "A's better" or "A and B could be the same" or "B's better." (It's unclear to me what would be the problem with looking at the P values for both the > test and the != test, for a single test, and only "falling back" to the > test if you happen to be in a range where you can say "we can't say for sure that B's better, but we can be pretty sure that A isn't better," like how the different P values are presented here[1].)
It still wouldn't be valid to do something like "oh the two-sided one was inconclusive but leaned in favor of A, so let's do a one-sided one to check if A's better after all," though.
[1] http://www.ats.ucla.edu/stat/mult_pkg/faq/general/tail_tests...