Re:Annoying A/B testing mistakes every engineer should know
by whimsicalism on Friday June 16, 2023 @12:11PM
(#36359051)
I don't follow. If one bucket has many more iOS users, it seems like you have done a bad job randomizing your treatment?
Re:Annoying A/B testing mistakes every engineer should know
by BoppreH on Friday June 16, 2023 @01:30PM
(#36360528)
It could be self-selection happening after you randomized the groups. For example a desktop landing page advertising an app, which might be installed on either mobile operating system.
Consider the case where iOS users are more likely to convert than Android users, but you currently have very few iOS users. You then A/B test a new design that imitates iOS, but has awful copy. Both iOS and Android users are less likely to convert, but it attracts more iOS users.
The group as a whole has higher conversion because of the demographic shift, but every subgroup has less.