News for hackers. Stuff that matters.
Re:A/B Testing Rigorously (without losing your job) by majormajor on Thursday January 24, 2013 @06:39PM (#5112995)
Wouldn't it just be a normal one-sided hypothesis test? The only thing that changes under H_0: A > B vs H_0: A = B is that you're only interested in the area under one side of the curve instead of both. The test statistic remains the same, you just get a different p value.

It should let you get away with a lower sample size at the expense of possibly only being able to conclude "A's better" or "A's not better" instead of "A's better" or "A and B could be the same" or "B's better." (It's unclear to me what would be the problem with looking at the P values for both the > test and the != test, for a single test, and only "falling back" to the > test if you happen to be in a range where you can say "we can't say for sure that B's better, but we can be pretty sure that A isn't better," like how the different P values are presented here[1].)

It still wouldn't be valid to do something like "oh the two-sided one was inconclusive but leaned in favor of A, so let's do a one-sided one to check if A's better after all," though.

[1] http://www.ats.ucla.edu/stat/mult_pkg/faq/general/tail_tests...

Re:A/B Testing Rigorously (without losing your job) by btilly on Thursday January 24, 2013 @07:18PM (#5113162)
In this article I was going for the highest possible standard. The ins and outs of one-sided versus two-sided was not a topic that I wanted to explore.

In the next article I plan to relax things a lot farther than just one-sided versus two-sided tests!

Re:A/B Testing Rigorously (without losing your job) by majormajor on Thursday January 24, 2013 @07:25PM (#5113187)
Gotcha, thanks. And one- vs two-sided alone would hardly be a huge win, I was just curious since I didn't remember seeing it brought up at all in any of the prior articles I'd read.