But the best I've seen was "I bet you can't hit me with a quarter".
I guess what I'm asking is this: Is A/B testing an evolutionary approach to developing a product? Are "revolutionary" leaps possible if you strictly follow the A/B testing methodology? (My guess: probably not, which implies that most people don't follow A/B exclusively, which sounds like common sense).
Of course, in theory, A/B as the name implies is about testing two different approaches, so again, in theory, any stuff completely different can be seen as an experiment for A/B. The problem is that humans have been trying different ways to do things for a loooooooooong time, and the whole point of trying to teach and learn A/B is to test things, measure, and walk towards your goal.
So yes, you can "A/B test" two completely different things, but it will be so. much. harder to actually know what it worked. For example, you're A/B testing a bicycle and a Ferrari Testarossa. D'uh... obviously they'll work differently, but what caused? Price, color, speed, brand?
That's why when A/B testing is mentioned it's about small, isolable, repeatable, incremental and evolutionary approach. Because it's a technique with this purpose in mind.
I was expecting something where he made iterative tweaks to the creative (i.e. the sign) and did continual comparisons of the success rate to see how different changes affected the result.
All this guy did was take a homeless guy's sign and add a really cool marketing idea to it, which increased the homeless guy's take. Yes, the results of his changes were awesome, but it really had nothing to do with "A/B testing" per se.