Re:A/B test improved your website's conversion rate? Not so ...
by ComputerGuru on Friday January 07, 2022 @05:56PM
(#29846061)
Not the person you're responding to, but yes, that's the idea. It's a control not for the the B but for the unknown unknowns that may or may not be there.
If A' and B both statistically differ from A, then you have a problem because you're not testing what you think you are testing, regardless of what your naive A/B test's p-value would have indicated.
If A' and B both statistically differ from A, then you have a problem because you're not testing what you think you are testing, regardless of what your naive A/B test's p-value would have indicated.