News for hackers. Stuff that matters.
Re:Most Winning A/B Test Results are Illusory [pdf] by AJ007 on Sunday February 23, 2014 @10:54PM (#7289003)
There are a lot of published "case studies" in the internet marketing field that consist of a few hundred views and a handful of conversions. It is even more embarrassing considering you often need 100,000+ unique visitors and thousands of conversions to find real winners.. and you still have to deal with a reality that a real 'winner' may result in a drop off of sales (in lead generation), an increase in charge backs (if your conversion was a sale), etc. This accounts for a sliver of the regression to the mean mentioned in the whitepaper.

Tests have value, but just making your site/app very simple and completely non-confusing to the viewer can do something years of split tests will not.

I suggest running tests and monitoring metrics as you implement design changes, not so much as a magic eight ball, but to ensure you avoid truly catastrophic UI fuck ups.

There are no comments on this comment yet.