Why multi-armed bandit algorithm is not "better" than A/B testing

Re:Why multi-armed bandit algorithm is not "better" than A/B... by arkitaip on Friday June 01, 2012 @09:51AM (#4053330)

It seems like most people who use these content optimization tools don't really understand the statistics involved. What are your thought on this? How do you educate your users on the merit of your approach vs a/b testing when the topic is so complex?

Also, despite this being a slightly pro a/b testing post, I have to say it's actually made me more interested in trying out Myna's approach MAB algorithm.

[ Parent | on HN ]

Re:Why multi-armed bandit algorithm is not "better" than A/B... by Drbble on Friday June 01, 2012 @10:19AM (#4053469)

Same way every product from GWO and T&T on down: show a pretty graph that ignores the underlying assumption that it's even possible to use statistics to conjure certainty from uncertainty, and trust that users will never know or care about the difference.

/former AB test software dev who fought my users to try to stop them from misinterpretation results, and failed.

[ Parent | on HN ]

Re:Why multi-armed bandit algorithm is not "better" than A/B... by btilly on Friday June 01, 2012 @11:09AM (#4053797)

If it gives you comfort, if there is a significant underlying difference and the calculations are done right, with high probability they will get the right answer even though they are misunderstanding the statistics.

Acceptance of this fact has avoided a lot of potential ulcers for me.

[ Parent | on HN ]

Re:Why multi-armed bandit algorithm is not "better" than A/B... by paraschopra on Friday June 01, 2012 @10:14AM (#4053446)

Just to be clear, we don't have anti-MAB stance or pro-A/B testing. The point was that MAB is not "better" as an earlier article titled (20 lines of code that beat A/B testing) had claimed. These methodologies clearly serve two different needs.

[ Parent | on HN ]