Don't get me wrong - A/B tests are valuable, but I've worked at many companies that have relied on them to a fault.

For one thing, there are very important metrics that are hard to measure, and many are impossible to measure in relation to a specific website change. User happiness for example is a nearly impossible metric to measure, and correlating it to specific website experiments is pretty much right out.

Many factors also have time-delayed effects, which is nearly impossible to correlate back to source causes, and IMO why we see so many companies achieve short-term conversion gains at enormous expense to brand perception and customer satisfaction.

In many ways companies that over-focus on A/B testing miss the forest for the trees - it doesn't necessarily create the best possible product, it often instead ends up creating the least shitty version of a bad product.