News for hackers. Stuff that matters.
Re:Annoying A/B testing mistakes every engineer should know by jameshart on Friday June 16, 2023 @09:17AM (#36356065)
Yes, I don’t think it’s possible to observe a simpson’s paradox in a simple conversion test, either.

Simpson’s paradox is about spurious correlations between variables - conversion analysis is pure Bayesian probability.

It shouldn’t be possible to have a group as a whole increase its probability to convert, while having every subgroup decrease its probability to convert - the aggregate has to be an average of the subgroup changes.

Re:Annoying A/B testing mistakes every engineer should know by BoppreH on Friday June 16, 2023 @11:03AM (#36357786)
Are you sure?

Consider the case where iOS users are more likely to convert than Android users, but you currently have very few iOS users. You then A/B test a new design that imitates iOS, but has awful copy. Both iOS and Android users are less likely to convert, but it attracts more iOS users.

The group as a whole has higher conversion because of the demographic shift, but every subgroup has less.

Re:Annoying A/B testing mistakes every engineer should know by whimsicalism on Friday June 16, 2023 @12:11PM (#36359051)
I don't follow. If one bucket has many more iOS users, it seems like you have done a bad job randomizing your treatment?
Re:Annoying A/B testing mistakes every engineer should know by BoppreH on Friday June 16, 2023 @01:30PM (#36360528)
It could be self-selection happening after you randomized the groups. For example a desktop landing page advertising an app, which might be installed on either mobile operating system.
Re:Annoying A/B testing mistakes every engineer should know by HWR_14 on Friday June 16, 2023 @12:34PM (#36359482)
Simpson's paradox is sometimes about spurious correlations, but the original paradox Simpson wrote about was simply a binary question with 84 subgroups, where 3 or 4 subgroups with the outlying answer just had a significant enough amount of all samples, and a significant enough effect, to mutate the whole.