Re:A/B testing original design vs. random template bought fr...
by no_gravity on Friday March 15, 2019 @08:31AM
(#19399059)
What you've tested using that invocation is the
null hypothesis that the underlying probability
of 39/67 is 0.5.
Isn't that equivalent to my interpretation of the test result? "In a world where it makes no difference which design is used, you would get a result as significant as this 22% of the time". If you want to perform a test of a difference of two proportions, you need to do:
prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions))
Do you mean c(39,28)? Because group_a had 39 hits and group_b hat 28. Doing so with the group sizes Bemmu stated (3000/3000) also gives me a p value of 0.22.As long as the group sizes are equal, the test is not very sensitive to the sizes.
Re:A/B testing original design vs. random template bought fr...
by muraiki on Friday March 15, 2019 @10:03AM
(#19399752)
I think there is a difference in the approaches, given that the Chi-squared test statistic for the two-sample version is ~1.52, while for your one-sample version it is ~1.81. If group size doesn't matter and if you're justified in adding up successes as you have, I'd expect the test statistics to be nearly the same.
Edit: I'd expect them to be nearly the same since the Chi-squared distributions would be parameterized similarly, so if we have similar results, we should see similar test statistics. Maybe my reasoning here is incorrect though!
If you want to perform a test of a difference of two proportions, you need to do:
prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions))
I don't have experience with A/B testing, so I'm not sure if this is typically or best handled using this particular statistical test.
Edit: The first parameter should be c(39, 28), meaning the total conversions in each group. I have no excuse beyond being tired.
Edit 2: To clarify, I think he should still use the two-sample form of prop.test, especially since we did not know at the time of his posting that the sample sizes are equal.