A/B testing original design vs. random template bought from ThemeForest

Re:A/B testing original design vs. random template bought fr... by no_gravity on Friday March 15, 2019 @08:31AM (#19399059)

    What you've tested using that invocation is the
    null hypothesis that the underlying probability
    of 39/67 is 0.5.

Isn't that equivalent to my interpretation of the test result? "In a world where it makes no difference which design is used, you would get a result as significant as this 22% of the time".

    If you want to perform a test of a difference of two proportions, you need to do:
    prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions))

Do you mean c(39,28)? Because group_a had 39 hits and group_b hat 28. Doing so with the group sizes Bemmu stated (3000/3000) also gives me a p value of 0.22.

As long as the group sizes are equal, the test is not very sensitive to the sizes.

[ Parent | on HN ]

Re:A/B testing original design vs. random template bought fr... by muraiki on Friday March 15, 2019 @10:03AM (#19399752)

I think there is a difference in the approaches, given that the Chi-squared test statistic for the two-sample version is ~1.52, while for your one-sample version it is ~1.81. If group size doesn't matter and if you're justified in adding up successes as you have, I'd expect the test statistics to be nearly the same.

Edit: I'd expect them to be nearly the same since the Chi-squared distributions would be parameterized similarly, so if we have similar results, we should see similar test statistics. Maybe my reasoning here is incorrect though!

[ Parent | on HN ]