News for hackers. Stuff that matters.
Re:Out of the cesspool and into the sewer: A/B testing trap by josefresco on Monday June 21, 2010 @11:58AM (#1449157)
The main issue I battle with with A/B testing is backing out far enough to know whether the change is related to the edits made or simply a seasonal/daily/hourly/freak change.

I manage CPC campaigns for clients, and often times I make A/B changes in the keywords/bids/ads and wait 30-60-90 days before measuring and making changes based on the findings. And even with 90 days and tens of thousands of visits, I find that sometimes my scope is too limited and I get nailed by seasonal/yearly changes that the numbers didn't reveal.

A/B testing in 5K chunks works if everything else in the universe remains constant, but here in the real world that's never the case.

The worst is when you make a change, measure results and implement said change, and then something related to the economy (or even Weather) horks your CAC and suddenly the client isn't so thrilled with your amazing A/B improvements.

There are no comments on this comment yet.