Re:A/B test improved your website's conversion rate? Not so ...
by pottertheotter on Friday January 07, 2022 @01:06PM
(#29842457)
For an A/A/B test are you taking three samples (instead of two), and two of the three get shown the same thing (A)? Then you only consider the results for the B group if the two A groups show the same behavior?
Re:A/B test improved your website's conversion rate? Not so ...
by ComputerGuru on Friday January 07, 2022 @05:56PM
(#29846061)
Not the person you're responding to, but yes, that's the idea. It's a control not for the the B but for the unknown unknowns that may or may not be there.
If A' and B both statistically differ from A, then you have a problem because you're not testing what you think you are testing, regardless of what your naive A/B test's p-value would have indicated.
Re:A/B test improved your website's conversion rate? Not so ...
by Maxion on Friday January 07, 2022 @06:08PM
(#29846226)
You take three samples and two of three get shown the same thing. What happens here is both A groups will show different results until your sample grows enough that the CI window becomes small.
This helps to show the effect of a low sample from a non-uniform distribution.
A lot of people (me included) think they know statistics, but they don't.
The blogpost in OP also tries to explain the same thing - you shouldn't do statistics without understanding what it is your doing.
Re:A/B test improved your website's conversion rate? Not so ...
by bananamansion on Saturday January 08, 2022 @09:55AM
(#29852202)
dont you then need to run the experiment for a long long time to get the significance of it? + the site needs enough users viewing the page to run it in any reasonable amount of time.. i would think most sites dont get enough traffic to run a/a let alone ab?
The reason I wanted this type of test was because it was a waste of time testing shades of blue or two headlines that only differed by 2 words. The test variants were never radical enough to see any kind of significant uplift. Then after 5-10 tests the design starts to suffer by wandering down some weird path that nobody would consciously design from the outset. But the series of test "winners" made things go off in wild directions.
I still think there is some value in A/B testing (A/A/B only, if I'm honest). But in a small team, it's a waste of time.