News for hackers. Stuff that matters.
Re:A/B test improved your website's conversion rate? Not so ... by pottertheotter on Friday January 07, 2022 @01:06PM (#29842457)
For an A/A/B test are you taking three samples (instead of two), and two of the three get shown the same thing (A)? Then you only consider the results for the B group if the two A groups show the same behavior?
Re:A/B test improved your website's conversion rate? Not so ... by ComputerGuru on Friday January 07, 2022 @05:56PM (#29846061)
Not the person you're responding to, but yes, that's the idea. It's a control not for the the B but for the unknown unknowns that may or may not be there.

If A' and B both statistically differ from A, then you have a problem because you're not testing what you think you are testing, regardless of what your naive A/B test's p-value would have indicated.

Re:A/B test improved your website's conversion rate? Not so ... by Maxion on Friday January 07, 2022 @06:08PM (#29846226)
You take three samples and two of three get shown the same thing. What happens here is both A groups will show different results until your sample grows enough that the CI window becomes small.

This helps to show the effect of a low sample from a non-uniform distribution.

A lot of people (me included) think they know statistics, but they don't.

The blogpost in OP also tries to explain the same thing - you shouldn't do statistics without understanding what it is your doing.

Re:A/B test improved your website's conversion rate? Not so ... by bananamansion on Saturday January 08, 2022 @09:55AM (#29852202)
dont you then need to run the experiment for a long long time to get the significance of it? + the site needs enough users viewing the page to run it in any reasonable amount of time.. i would think most sites dont get enough traffic to run a/a let alone ab?