News for hackers. Stuff that matters.

A comment on 20 lines of code that beat A/B testing every time. Parent comment

Re:20 lines of code that beat A/B testing every time by timr on Wednesday May 30, 2012 @03:22PM (#4044378)
The draws aren't independent. At any given time, the probability of assigning a user to a cohort is dependent upon a function of the previous observations (in other words, it's a markov model).

The standard confidence tests -- t-tests, G-tests, chi-squared tests, etc. -- based on distributions of independent, identically distributed (iid) data.

I'd have to think about it more, but I believe that btilly's examples are also the most intuitive reasons why independence matters. If your data is time-dependent, then assigning users to cohorts based on past performance lets the time dependency dominate. There may be other good examples.

[ Parent | on HN ]
Re:20 lines of code that beat A/B testing every time by conductrics on Thursday May 31, 2012 @06:23AM (#4047298)
Is that true in the e-greedy case? Sure, during the exploit call, they are not independent, but during the explore portion I would assume they are, since they have been randomly assigned into the exploration pool (epsilon) and then drawn from a uniform random draw. There is no information that I can see from prior draws being used.
[ Parent | on HN ]