Re:A/B test improved your website's conversion rate? Not so ...
by pottertheotter on Friday January 07, 2022 @01:06PM
(#29842457)
For an A/A/B test are you taking three samples (instead of two), and two of the three get shown the same thing (A)? Then you only consider the results for the B group if the two A groups show the same behavior?
[ Parent
| on HN ]
Re:A/B test improved your website's conversion rate? Not so ...
by ComputerGuru on Friday January 07, 2022 @05:56PM
(#29846061)
Not the person you're responding to, but yes, that's the idea. It's a control not for the the B but for the unknown unknowns that may or may not be there.
If A' and B both statistically differ from A, then you have a problem because you're not testing what you think you are testing, regardless of what your naive A/B test's p-value would have indicated.
Re:A/B test improved your website's conversion rate? Not so ...
by Maxion on Friday January 07, 2022 @06:08PM
(#29846226)
You take three samples and two of three get shown the same thing. What happens here is both A groups will show different results until your sample grows enough that the CI window becomes small.
This helps to show the effect of a low sample from a non-uniform distribution.
A lot of people (me included) think they know statistics, but they don't.
The blogpost in OP also tries to explain the same thing - you shouldn't do statistics without understanding what it is your doing.
Re:A/B test improved your website's conversion rate? Not so ...
by bananamansion on Saturday January 08, 2022 @09:55AM
(#29852202)
dont you then need to run the experiment for a long long time to get the significance of it? + the site needs enough users viewing the page to run it in any reasonable amount of time.. i would think most sites dont get enough traffic to run a/a let alone ab?