Re:Annoying A/B testing mistakes every engineer should know
by plagiarist on Friday June 16, 2023 @12:39PM
(#36359578)
That doesn't test noticing the button, that tests clicking the button. If the color changes it is possible that fewer people notice it but are more likely to click in a way that increases total traffic. Or more people notice it but are less likely to click in a way that reduces traffic.
We see a lot of ghosts in A/B testing because we are loosey goosey about our denominators. Mathematicians apparently hate it when we do that.