The Imperfect World of A/B Testing Selection

Re:The Imperfect World of A/B Testing Selection by btilly on Thursday December 03, 2015 @04:49PM (#10672926)

You have to know the limitations of the approach you are using.

Also about session cookies, there is no correlation created if the A/B test behavior is tied to the session. The downside is that different users get different behaviors on different days. This may be a bad user experience. The upside is that it is quick and simple for things like landing pages.

In the end there is no solution that avoids actually understanding what your data really says.

[ Parent | on HN ]

Re:The Imperfect World of A/B Testing Selection by yummyfajitas on Thursday December 03, 2015 @06:41PM (#10673583)

There is absolutely correlation between sessions. If visitor 1 (corresponding to sessions 1,2,3) has a high conversion probability, while visitor 2 (corresponding to sessions 4,5,6) has a low conversion probability, then you've introduced correlation between sessions 1,2,3 and sessions 4,5,6. This breaks the CLT and all the usual independence assumptions.

If most of your visitors only have one session this may not matter...but then again with only session cookies you don't even have a way to know this.

[ Parent | on HN ]