News for hackers. Stuff that matters.
Re:The multiple comparisons problem by rfergie on Wednesday January 07, 2015 @11:36AM (#8851463)
You should look at multi-armed bandits - this approach is pretty much what you have reached in your conclusion and a lot of work has been done on them
Re:The multiple comparisons problem by noelwelsh on Wednesday January 07, 2015 @11:59AM (#8851618)
Funnily enough, my starting point to get into A/B testing was research on multiarmed bandits.

Update: I should add I think the issues are more complex than simply using, say, Thompson sampling.

Re:The multiple comparisons problem by rfergie on Wednesday January 07, 2015 @12:18PM (#8851750)
Ha! I should have read your profile before responding.