News for hackers. Stuff that matters.
Re:Appsumo reveals its A/B testing secret: only 1 out of 8 t... by ig1 on Tuesday February 15, 2011 @06:37AM (#2220875)
These aren't failures, if the A/B test shows no impact or that the change is a negative, then both of those add value in telling you what to concentrate on.

Before people did A/B test, it was generally assumed on Facebook ads that the title was the second most important factor on CTR so people spent a lot of time tuning title copy.

After lots of people did A/B testing it turns out that title copy has almost no impact, people switched their title to Chinese (for a non-chinese speaking audience) without seeing any change. That "no significant difference" means that thousands of man hours can be spent elsewhere rather than copywriting ad titles.

Re:Appsumo reveals its A/B testing secret: only 1 out of 8 t... by paraschopra on Tuesday February 15, 2011 @06:42AM (#2220881)
I'd be wary of concluding too much from other people's A/B tests. Yes, they serve as a good starting point but doing A/B tests yourself is still imperative. Case in point: Noah found out headlines didn't matter to his audience. But for many of our customers, a small change in headline does produce significant results.
Re:Appsumo reveals its A/B testing secret: only 1 out of 8 t... by ig1 on Tuesday February 15, 2011 @09:10AM (#2221212)
Sure, I wasn't saying that the headline results were generalizable, but "no significant difference" A/B testing results can tell you to stop wasting time in an area.