News for hackers. Stuff that matters.

A comment on 20 lines of code that beat A/B testing every time. Parent comment

Re:20 lines of code that beat A/B testing every time by pilom on Tuesday May 29, 2012 @10:10PM (#4041012)
I got a very different impression of this method than you did because I saw it as, just as with an A/B test, when you make a major change to the site, you reset your counters. This makes your point #2 mute. As for number 5, why couldn't you use the same cohort analysis with this method?
[ Parent | on HN ]
Re:20 lines of code that beat A/B testing every time by btilly on Wednesday May 30, 2012 @12:49AM (#4041429)
What is a major change?

I've seen minor tweaks to a form raise conversions by 20%. The person running a particular test may not even know that a particular change was significant, or even that it happened. The change could be as subtle as another running test realized that version x is better, interfering with existing tests.

As for #5, with an A/B test you run into these situations, you're able to break down and crunch the numbers in multiple ways, and then have a discussion about how you want to proceed. But with a multi-armed bandit approach whatever complexities you have not thought of and baked into your approach, are not going to be noticed.

[ Parent | on HN ]
Re:20 lines of code that beat A/B testing every time by ramses0 on Wednesday May 30, 2012 @08:18AM (#4042432)
moot.

mute == "does not talk"

moot == "of little or no practical value or meaning; purely academic."

moot/mute is a tricky word combo because smart people can justify using "MUTE" instead of moot, and english is such a terrible language w.r.t. spelling there are no good clues to use one versus the other (and "moot" is far less commonly used).

[ Parent | on HN ]
Re:20 lines of code that beat A/B testing every time by pooriaazimi on Wednesday May 30, 2012 @08:50AM (#4042535)
Off-topic: Most of the time, such grammar/spelling corrections get downvoted because people think they are nitpicking and don't add anything to the conversation, or are rude somehow, but as a non-English, I appreciate them a lot!
[ Parent | on HN ]