News for hackers. Stuff that matters.

A comment on Annoying A/B testing mistakes every engineer should know. Parent comment

Re:Annoying A/B testing mistakes every engineer should know by datastoat on Friday June 16, 2023 @10:58AM (#36357693)
That xkcd comic highlights the problem with observational (as opposed to controlled) studies. TFA is about A/B testing, i.e. controlled studies. It’s the fact that you (the investigator) is controlling the treatment assignment that allows you to draw causal conclusions. What you happen to believe about the mechanism of action doesn’t matter, at least as far as the outcome of this particular experiment is concerned. Of course, your conjectured mechanism of action is likely to matter for what you decide to investigate next.

Also, frequentism / Bayesianism is orthogonal to causal / correlational interpretations.

[ Parent | on HN ]
Re:Annoying A/B testing mistakes every engineer should know by carlmr on Friday June 16, 2023 @12:17PM (#36359163)
I think what kevinwang is getting at, is that if you A/B test with a static version A and enough versions of B, at some point you will get statistically significant results if you repeat it often enough.

Having a control doesn't mean you can't fall victim to this.

[ Parent | on HN ]
Re:Annoying A/B testing mistakes every engineer should know by ricardobeat on Friday June 16, 2023 @06:20PM (#36364737)
You control statistical power and the error rate, and choose to accept a % of false results.
[ Parent | on HN ]
Re:Annoying A/B testing mistakes every engineer should know by majormajor on Friday June 16, 2023 @02:25PM (#36361390)
AB tests are still vulnerable to p-hacking-esque things (though usually unintentional). Run enough of them and your p value is gonna come up by chance sometimes.

Observational ones are particularly prone because you can slice and dice the world into near-infinite observation combinations, but people often do that with AB tests too. Shotgun approach, test a bunch of approaches until something works, but if you'd run each of those tests for different significance levels, or for twice as long, or half as long, you could very well see the "working" one fail and a "failing" one work.

[ Parent | on HN ]
Re:Annoying A/B testing mistakes every engineer should know by eVoLInTHRo on Friday June 16, 2023 @02:46PM (#36361761)
The xkcd comic seems more about the multiple comparisons problem (https://en.wikipedia.org/wiki/Multiple_comparisons_problem), which could arise in both an observational or controlled setting.
[ Parent | on HN ]