News for hackers. Stuff that matters.

A comment on A/B testing mistakes I learned the hard way. Parent comment

Re:A/B testing mistakes I learned the hard way by gregbarbosa on Monday August 19, 2024 @12:05PM (#41292198)
> Simpson's paradox is when more button presses lead to more purchases. But then you look at desktop vs mobile and you find out that for both desktop and mobile more clicks doesn't mean more purchases (or worse, more clicks means fewer purchases).

How could more button presses lead to increased conversion rates while hiding this data when comparing desktop and mobile? Wouldn’t you see at least one device type demonstrating higher CVR to reflect aggregate CVR increase?

[ Parent | on HN ]
Re:A/B testing mistakes I learned the hard way by light_hue_1 on Friday August 30, 2024 @01:11PM (#41402644)
That's Simpson's paradox!

You can take data where as a whole presses lead to more purchases. Then split it into two halves (like mobile vs desktop) and show that on both halves presses lead to fewer purchases.

The whole paradox is that the intuition we have for averages doesn't apply to correlations.

I suggest checking out the Wikipedia page.

[ Parent | on HN ]