News for hackers. Stuff that matters.
Re:A/B testing original design vs. random template bought fr... by closed on Friday March 15, 2019 @05:37PM (#19404551)
I don't think that's the correct use of prop.test for this question. When you give it two numbers (success, total), it tests against the null that chance of success is 50%.

Here, we want to test whether p(success|cond) differs across conditions, not whether prob(cond|success) is 50%.

This distinction is important because when p(success|cond) for some cond is low, its variance is also very small, but prop.test(39,67) doesn't reflect this. That could be 67 successes out of a small sample (and high chance of success), or out of a huge sample (and low chance of success).

Edit: whoops, I didn't notice other comments point out this issue

There are no comments on this comment yet.