Re:Making UX Choices Based on What You Hate About the Internet
by potatolicious on Tuesday May 17, 2016 @02:08PM
(#11715872)
I used to be a firm believer in A/B tests, but nowadays I feel that our over-reliance on it has created this awful internet dystopia.
Don't get me wrong - A/B tests are valuable, but I've worked at many companies that have relied on them to a fault.
For one thing, there are very important metrics that are hard to measure, and many are impossible to measure in relation to a specific website change. User happiness for example is a nearly impossible metric to measure, and correlating it to specific website experiments is pretty much right out.
Many factors also have time-delayed effects, which is nearly impossible to correlate back to source causes, and IMO why we see so many companies achieve short-term conversion gains at enormous expense to brand perception and customer satisfaction.
In many ways companies that over-focus on A/B testing miss the forest for the trees - it doesn't necessarily create the best possible product, it often instead ends up creating the least shitty version of a bad product.
Don't get me wrong - A/B tests are valuable, but I've worked at many companies that have relied on them to a fault.
For one thing, there are very important metrics that are hard to measure, and many are impossible to measure in relation to a specific website change. User happiness for example is a nearly impossible metric to measure, and correlating it to specific website experiments is pretty much right out.
Many factors also have time-delayed effects, which is nearly impossible to correlate back to source causes, and IMO why we see so many companies achieve short-term conversion gains at enormous expense to brand perception and customer satisfaction.
In many ways companies that over-focus on A/B testing miss the forest for the trees - it doesn't necessarily create the best possible product, it often instead ends up creating the least shitty version of a bad product.