News for hackers. Stuff that matters.
Re:A/B testing and the historic lift paradox by wodenokoto on Wednesday July 28, 2021 @05:41AM (#27981943)
I’ve never done A/B testing, but is it normal to have the control group and the treatment group historically be so different?

The historical difference in average is much larger than the after treatment. Can those groups even be compared after treatment?

Re:A/B testing and the historic lift paradox by pkoperek on Thursday July 29, 2021 @06:46AM (#27994943)
I think the point is that the control group and treatment groups are in fact the same group (you were giving them the same "service" before starting the experiment) and any difference in the beta_C and beta_T is just random noise. The key part of the argument is that if N goes to infinity the beta_C should be equalt to beta_T (hence it doesn't make much sense to compare them).