Re:A/B testing original design vs. random template bought fr...
by closed on Friday March 15, 2019 @05:37PM
(#19404551)
I don't think that's the correct use of prop.test for this question. When you give it two numbers (success, total), it tests against the null that chance of success is 50%.
Here, we want to test whether p(success|cond) differs across conditions, not whether prob(cond|success) is 50%.
This distinction is important because when p(success|cond) for some cond is low, its variance is also very small, but prop.test(39,67) doesn't reflect this. That could be 67 successes out of a small sample (and high chance of success), or out of a huge sample (and low chance of success).
Edit: whoops, I didn't notice other comments point out this issue
Here, we want to test whether p(success|cond) differs across conditions, not whether prob(cond|success) is 50%.
This distinction is important because when p(success|cond) for some cond is low, its variance is also very small, but prop.test(39,67) doesn't reflect this. That could be 67 successes out of a small sample (and high chance of success), or out of a huge sample (and low chance of success).
Edit: whoops, I didn't notice other comments point out this issue