News for hackers. Stuff that matters.
Re:Marketing Lessons from A/B Testing a Homeless Man's Sign by Splines on Monday March 29, 2010 @05:21PM (#1227585)
Would A/B testing encompass changes as large as what happened here? Having never done A/B testing, it sounds like an iterative process. Would continuous A/B testing ever converge on a solution like the one arrived here?

I guess what I'm asking is this: Is A/B testing an evolutionary approach to developing a product? Are "revolutionary" leaps possible if you strictly follow the A/B testing methodology? (My guess: probably not, which implies that most people don't follow A/B exclusively, which sounds like common sense).

Re:Marketing Lessons from A/B Testing a Homeless Man's Sign by inerte on Monday March 29, 2010 @06:40PM (#1227745)
Yes, but not encouraged. The literature, or most common applications of A/B testing, is about little improvements. Phrases being changed, picture alignment, that kinda of thing.

Of course, in theory, A/B as the name implies is about testing two different approaches, so again, in theory, any stuff completely different can be seen as an experiment for A/B. The problem is that humans have been trying different ways to do things for a loooooooooong time, and the whole point of trying to teach and learn A/B is to test things, measure, and walk towards your goal.

So yes, you can "A/B test" two completely different things, but it will be so. much. harder to actually know what it worked. For example, you're A/B testing a bicycle and a Ferrari Testarossa. D'uh... obviously they'll work differently, but what caused? Price, color, speed, brand?

That's why when A/B testing is mentioned it's about small, isolable, repeatable, incremental and evolutionary approach. Because it's a technique with this purpose in mind.