It is easy to believe the things you want to believe. A/B testing is useful for convincing you of the things you don't want to believe.

Example: I'm releasing my Rails A/B testing framework this Sunday, with a case study writeup taken from my site. My site currently allows people to defer signup for the trial by means of a guest login. That sort of option is popular here. I'm told it increases usability and user engagement, right?

Without ruining the surprise: that is a testable hypothesis. ;)

A/B testing is great, no doubt. The lesson of solely simplifying is just not a great outcome of such a test. It's like driving a car for the first time and then telling everybody that it's faster than walking.

Yeah you do. I ran a test on a friend's site, convinced that a short version would outperform the long-copy version. Conversions dropped by about 30%, which would have cost him tens of thousands of dollars per year.