News for hackers. Stuff that matters.

A comment on 20 lines of code that beat A/B testing (2012). Parent comment

Re:20 lines of code that beat A/B testing (2012) by teekert on Wednesday April 06, 2016 @05:06AM (#11437572)
" If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc."

I'd serve them at the same time, randomize which clients see which one.

[ Parent | on HN ]
Re:20 lines of code that beat A/B testing (2012) by spiderfarmer on Wednesday April 06, 2016 @07:57AM (#11438126)
Ofcourse, that's what A/B testing is about. But this only works if you have lots of traffic. The smaller the amount of traffic, the bigger the influence of offsite factors. Also, the results become more difficult to explain: https://news.ycombinator.com/item?id=11438098
[ Parent | on HN ]
Re:20 lines of code that beat A/B testing (2012) by chii on Wednesday April 06, 2016 @05:45AM (#11437719)
And it turns out you randomly assigned B to a poorer user more often than not, and this make B worse...
[ Parent | on HN ]
Re:20 lines of code that beat A/B testing (2012) by eru on Wednesday April 06, 2016 @06:12AM (#11437801)
That's why you do it properly randomized. Statistics tell you how much of a risk of your scenario (and similar) you are running at most.
[ Parent | on HN ]