News for hackers. Stuff that matters.
Re:Annoying A/B testing mistakes every engineer should know by travisjungroth on Friday June 16, 2023 @10:32AM (#36357151)
The biggest issue with those three hypotheses is one of them, the noticing the button, almost certainly isn't being tested. But, however the test goes will inform how people think about that hypothesis.
Re:Annoying A/B testing mistakes every engineer should know by hinkley on Friday June 16, 2023 @11:42AM (#36358526)
Rate of traffic on the checkout page, divided by overall traffic.

We see a lot of ghosts in A/B testing because we are loosey goosey about our denominators. Mathematicians apparently hate it when we do that.

Re:Annoying A/B testing mistakes every engineer should know by plagiarist on Friday June 16, 2023 @12:39PM (#36359578)
That doesn't test noticing the button, that tests clicking the button. If the color changes it is possible that fewer people notice it but are more likely to click in a way that increases total traffic. Or more people notice it but are less likely to click in a way that reduces traffic.
Re:Annoying A/B testing mistakes every engineer should know by ano-ther on Friday June 16, 2023 @05:26PM (#36364131)
Good observation that the noticing doesn’t get tested.

Would there be any benefit from knowing the notice rate though? After all, the intended outcome is increased sales by clicking.

Re:Annoying A/B testing mistakes every engineer should know by alsiola on Friday June 16, 2023 @05:57PM (#36364478)
This is what I was driving at in my original comment - the intermediary steps are not of interest (from the POV of the hypothesis/overall experiment), so why mention them at all.
Re:Annoying A/B testing mistakes every engineer should know by ricardobeat on Friday June 16, 2023 @06:10PM (#36364613)
Probably not, but then that hypothesis should not be part of the experiment.