News for hackers. Stuff that matters.

A comment on Annoying A/B testing mistakes every engineer should know. Parent comment

Re:Annoying A/B testing mistakes every engineer should know by carlmr on Friday June 16, 2023 @12:17PM (#36359163)
I think what kevinwang is getting at, is that if you A/B test with a static version A and enough versions of B, at some point you will get statistically significant results if you repeat it often enough.

Having a control doesn't mean you can't fall victim to this.

[ Parent | on HN ]
Re:Annoying A/B testing mistakes every engineer should know by ricardobeat on Friday June 16, 2023 @06:20PM (#36364737)
You control statistical power and the error rate, and choose to accept a % of false results.
[ Parent | on HN ]