News for hackers. Stuff that matters.
Re:A/B testing original design vs. random template bought fr... by muraiki on Friday March 15, 2019 @08:08AM (#19398897)
That's not how you use prop.test. What you've tested using that invocation is the null hypothesis that the underlying probability of 39/67 is 0.5.

If you want to perform a test of a difference of two proportions, you need to do:

prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions))

I don't have experience with A/B testing, so I'm not sure if this is typically or best handled using this particular statistical test.

Edit: The first parameter should be c(39, 28), meaning the total conversions in each group. I have no excuse beyond being tired.

Edit 2: To clarify, I think he should still use the two-sample form of prop.test, especially since we did not know at the time of his posting that the sample sizes are equal.

Re:A/B testing original design vs. random template bought fr... by no_gravity on Friday March 15, 2019 @08:31AM (#19399059)

    What you've tested using that invocation is the
    null hypothesis that the underlying probability
    of 39/67 is 0.5.
Isn't that equivalent to my interpretation of the test result? "In a world where it makes no difference which design is used, you would get a result as significant as this 22% of the time".

    If you want to perform a test of a difference of two proportions, you need to do:
    prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions))
Do you mean c(39,28)? Because group_a had 39 hits and group_b hat 28. Doing so with the group sizes Bemmu stated (3000/3000) also gives me a p value of 0.22.

As long as the group sizes are equal, the test is not very sensitive to the sizes.

Re:A/B testing original design vs. random template bought fr... by muraiki on Friday March 15, 2019 @10:03AM (#19399752)
I think there is a difference in the approaches, given that the Chi-squared test statistic for the two-sample version is ~1.52, while for your one-sample version it is ~1.81. If group size doesn't matter and if you're justified in adding up successes as you have, I'd expect the test statistics to be nearly the same.

Edit: I'd expect them to be nearly the same since the Chi-squared distributions would be parameterized similarly, so if we have similar results, we should see similar test statistics. Maybe my reasoning here is incorrect though!