How Split-Testing Our Opt-In Form Increased Our Conversion Rate by 102.2%

Re:How Split-Testing Our Opt-In Form Increased Our Conversio... by bermanoid on Thursday February 03, 2011 @01:52PM (#2176111)

Never used Visual Website Optimizer, but chances are, whatever they do leaves you vulnerable to the flaw mentioned in the article that swalberg linked.

Basically, every time you peek at the results of a split test to see if the confidence level is high enough, you make your split test more prone to error.

There are ways to adjust for this which allow you to continuously monitor the test and stop if it hits a threshold, but it's much easier just to set a known number of samples to take and accept the confidence level that comes out then.

[ Parent | on HN ]

Re:How Split-Testing Our Opt-In Form Increased Our Conversio... by Datasta on Thursday February 03, 2011 @02:05PM (#2176205)

Actually, I had planned to test 2,000 views from the outset. It went a little over because I didn't stop it fast enough. I do agree that you are vulnerable to influencing results if you stop the test as soon as you hit your confidence level

[ Parent | on HN ]