Re:Why designers don’t like A/B testing
by redthrowaway on Monday August 08, 2011 @06:30AM
(#2859510)
It seems that the author isn't disaffected with A/B testing, but rather with improper implementations thereof. A/B testing is intended to be scientific, so it must therefore adhere to rigorous scientific principles of validity. Sure, there are those who implement it incorrectly, but a properly implemented A/B test should always have a (large) control group, against whom you can measure the purported improvements. Such complaints as A/B testing inherently fostering incrementalism that leads away from proper design can be countered by implementing a "proper" control at each stage. Similarly, his complaint of A/B testing being a binary proposition ignores the 0/1/many axiom of good engineering. An A/B test that forces the user to choose between two unattractive options is a stupid test. An A/B test that shows many options to be unattractive is a good test with a dissatisfying result.
Rather than question the underlying validity of the methodology, the author would be better served by questioning the validity of its implementation in the instances he's encountered.
Rather than question the underlying validity of the methodology, the author would be better served by questioning the validity of its implementation in the instances he's encountered.