Re:Why designers don’t like A/B testing
by kelnos on Monday August 08, 2011 @11:07AM
(#2860279)
Users are not always the best judge of high quality design. That’s why you hire expert designers of seasoned skills, experience, judgment, and yes the conviction to make a call as to what’s better overall.
In the end, aren't the users the only judge who matter? You hire expert designers to create an interface that your users judge to be pleasing, but also intuitive and easy to use. The user's judgement is king. If your users don't like your interface, it's game over. It doesn't matter how "cohesive" your designer thinks it is.
How do you A/B test novel interaction models, conceptual paradigms, visual styles (by the way, visuals & interactions have a two-way rapport, they inform each other, can’t separate them–see Mike Kruzeniski’s talks) which may vary wildly from before? Would you A/B test the Wii or Dyson or Prius or iPhone? Against what???
Rephrased: "I don't like A/B testing because you can't test everything using A/B testing." Huh?
A/B testing locks you into just two comparative options, an exclusively binary (and thus limited) way of thinking. What about C or D or Z or some other alternatives?
C'mon. The "A/B" in "A/B testing" isn't meant to limit choices. It's just a convenient way of expressing the idea. There's nothing stopping you from doing "A/B/C/D testing", or even starting out with 10 different ideas, and then using other methods to narrow them down to a few choices that you can then A/B test.
What if there are elements of A & B that could blend together to form another option? Avenues for generative design options are shut down by looking at only A and only B.
Sure, if you think of it in a ridiculously restrictive way. So do an A/B/C test between A, B, and some A+B blend that you think might work. Or make it an iterative process: do an A/B test, which let's say shows you B is better. Then do another A/B test, with the original B pitted against a B with some elements of A blended in.
If you look at A/B testing as the be-all-end-all and consider it the only tool in your arsenal, of course you're going to get bad results. But it has its uses in some places, and any designer who refuses to do A/B testing at all just seems uninformed and unwilling to examine hard data about how an interface might work.
In the end, aren't the users the only judge who matter? You hire expert designers to create an interface that your users judge to be pleasing, but also intuitive and easy to use. The user's judgement is king. If your users don't like your interface, it's game over. It doesn't matter how "cohesive" your designer thinks it is.
How do you A/B test novel interaction models, conceptual paradigms, visual styles (by the way, visuals & interactions have a two-way rapport, they inform each other, can’t separate them–see Mike Kruzeniski’s talks) which may vary wildly from before? Would you A/B test the Wii or Dyson or Prius or iPhone? Against what???
Rephrased: "I don't like A/B testing because you can't test everything using A/B testing." Huh?
A/B testing locks you into just two comparative options, an exclusively binary (and thus limited) way of thinking. What about C or D or Z or some other alternatives?
C'mon. The "A/B" in "A/B testing" isn't meant to limit choices. It's just a convenient way of expressing the idea. There's nothing stopping you from doing "A/B/C/D testing", or even starting out with 10 different ideas, and then using other methods to narrow them down to a few choices that you can then A/B test.
What if there are elements of A & B that could blend together to form another option? Avenues for generative design options are shut down by looking at only A and only B.
Sure, if you think of it in a ridiculously restrictive way. So do an A/B/C test between A, B, and some A+B blend that you think might work. Or make it an iterative process: do an A/B test, which let's say shows you B is better. Then do another A/B test, with the original B pitted against a B with some elements of A blended in.
If you look at A/B testing as the be-all-end-all and consider it the only tool in your arsenal, of course you're going to get bad results. But it has its uses in some places, and any designer who refuses to do A/B testing at all just seems uninformed and unwilling to examine hard data about how an interface might work.