News for hackers. Stuff that matters.

A comment on Lines of code that will beat A/B testing every time (2012). Parent comment

Re:Lines of code that will beat A/B testing every time (2012) by sweezyjeezy on Monday January 13, 2025 @10:43PM (#42693341)
Yes but here's a exaggerated version - say were to sample for a week at 50/50 when the base conversion rate was at 4%, then we sample at 25/75 for a week with the base conversion rate bumped up to 8% due to a sale.

The average base rate for the first variant is 5.3%, the second is 6.4%. Generally the favoured variant's average will shift faster because we are sampling it more.

[ Parent | on HN ]
Re:Lines of code that will beat A/B testing every time (2012) by necovek on Tuesday January 14, 2025 @08:14AM (#42696753)
Uhm, this still sounds like just bad math.

While it's non-obvious this is the effect, anyone analyzing the results should be aware of it and should only compare weighted averages, or per distinct time periods.

And therein is the largest problem with A/B testing: it's mostly done by people not understanding the math subtleties, thus they will misinterpret results in either direction.

[ Parent | on HN ]
Re:Lines of code that will beat A/B testing every time (2012) by sweezyjeezy on Tuesday January 14, 2025 @05:19PM (#42704725)
Agreed, and articles like this don't help. That's the only point I was trying to make really.
[ Parent | on HN ]