Re:Why A/B testing will loose against advanced AI, all the time
by olavolav on Friday March 17, 2017 @12:38PM
(#13895386)
Regarding that last bit: You will clearly get better results by using A/B tests than by having humans take a guess as to what works best – but as you already pointed out, those are not the only two possibilities. Different varieties of mutli-armed/contextual/etc. bandits, and then the new wave of reinforcement learning tricks (i.e. AI) are much better suited to the problem.
(Full disclosure: I didn't write the article, but I am the CTO of the company that the article is about.)
The idea of A/B testing is that humans are not good judging which version is better, so the only solution is to run an experiment, so in many cases you will get better results with an A/B test.
And there are also variations where the split is not constant, like the mutiarmed bandit. If A/B test is better or worse than multiarmed bandit is also a good flamewar topic.