This is the strangest warning against A/B tests I've heard. "Be careful because you might end up producing something that is #1 internet retailer". The Amazon home page might not be as sexy as the latest whizz-bang startup, but success doesn't lie.

Thank you. I can't count the times I've tried to explain this to people. The purpose of amazon's site is not that visitors think it's pretty, or even easy to use. It's to get you to buy as much as possible. And there's convincing evidence that it's quite successful in fulfilling that purpose.

Why do supermarkets put the most frequently sold items at the back, when it would be so much easier for customers if they were right by the entrance?

Indeed, not stepping back can lead you (via A/B testing) to just find a local maxima in usability/uptake/etc.

I don't particularly like Amazon's site either, I think it's cluttered, but it has improved a bit. It's still not the prettiest thing, but what it does right, though, is it makes it very, very easy to buy things. And that's all it needs to do.

But you're right about not always choosing the "less bad" when there are other options you may not be considering.

The problem is though that this observation is often used as an excuse to ignore data or avoid doing small iterative experiments. In most cases, this is a mistake. In other words, you could do a lot worse than reproducing Amazon.com.

Are you suggesting Amazon's practices have led it astray somehow?

It's just a pointed way of saying that an important part of testing is knowing what to test.