News for hackers. Stuff that matters.
Re:A/B Testing vs MAB algorithms - It's complicated by btilly on Monday September 17, 2012 @01:51PM (#4533811)
The win from actually testing is ridiculously more important than the marginal differences between different ways of running tests. Even if you later find out that your testing methodology was wrong, it is still a likely win.

If you've not already been convinced of that, there is no more important point that I can make to you about testing. And, sadly, most companies out there do not seem to have realized this point yet. But continually repeating it is boring for me, and useless for people who've already got that point.

That said, I would add a third case to your list where MAB is helpful. If you're clever with a MAB system, you can manage to roll new versions into the testing mix without ruining an ongoing test that you don't yet have final statistics on. This flexibility can be very helpful.

There are no comments on this comment yet.