News for hackers. Stuff that matters.
Re:How Coderwall grew over 50% with an A/B test by btilly on Tuesday September 27, 2016 @12:16PM (#12590722)
I was not involved in this test, but here is how I would have done this.

1. I would have randomly assigned pages with more than X average conversions to the versions, and then kept a count of how many pages saw an increase or a decrease.

2. The key phrase is, "If your traffic can support it." This is a question of doing statistical calculations in advance and deciding how big a random mistake you'd be willing to accept if you were unlucky. I find that people tend to err on the side of trying to get too much information out of too little data, so I would generally push towards fewer versions, not more.

3. I strongly disagree with you on this. I would prefer to run 5 small tests over one big one because you learn more. In particular I have found many times that titles, subject lines, and the like can have a surprising impact. The area of your site with the most likely big wins from A/B tests is where you have large numbers of unengaged visitors that you are trying to get to make the snap decision to click - which is exactly what this title test measures.

4. They are trying to run a business, not conduct a research product. Once they had a 14% known win on their hands, and enough sample data to be certain of it, there is no business value in continuing to do what you know is less effective. Do as they did. Collect overall figures, say "some fraction of that is our test", and don't try to measure exactly how big a fraction it is.

But to those points I would add a fifth. I have found that there is a significant "novelty" effect. For example if you're carrying on an email campaign, people will initially respond well to pretty much any subject line change the first time they see it. But not so much 3 weeks later. I doubt that was an issue here, but it is a problem to always keep in the back of your mind as a question.

There are no comments on this comment yet.