News for hackers. Stuff that matters.
Re:The multiple comparisons problem by trjordan on Wednesday January 07, 2015 @11:07AM (#8851286)
In many cases, I think this is fine.

Take the case given in the article: 2 designs that appear equal, but segmenting reveals (incorrectly) that one performs better on Android and the other on iPhone. Statistically, it doesn't matter which one you pick, because you don't have a strong result. Anecdotally, you might go down the wrong path and waste some time doing something like showing design A to Android and B to iPhone. But the only thing lost is time -- probably a small amount of time compared to creating the different designs.

Best case, you realize there's something technically wrong with your changes in a certain segment because things are widely different. Worst case, you make some changes that don't matter. Overall, the risk / reward of segmenting seems worthwhile.

Re:The multiple comparisons problem by yummyfajitas on Wednesday January 07, 2015 @12:15PM (#8851728)
This is very true - my post is mainly coming from a classical frequentist hypothesis testing perspective. From a Bayesian/portfolio optimization perspective, all that segmentation isn't hurting you much.

But from a business perspective, it's a huge distraction. Your marketers shouldn't be wasting time segmenting - there is no reason to believe it's making any money. And due to multiple comparisons, it's virtually guaranteed that your A/B tests will steer the marketers in the direction of more segmentation.