News for hackers. Stuff that matters.
Re:Ask HN: What is your one tip about A/B testing you would ... by Someone1234 on Thursday January 22, 2015 @11:09AM (#8929866)
That most A/B tests don't prove what they aim to prove. People just add a new version of the site, then wait until 100 people have seen it, measure a small improvement and migrate to the new version.

However if you then run the test a second time, but make the "old" site the alternative, you might find it too sees a small measure of improvement and thus you could bounce back and forth between A and B until the end of time.

I am not a statistician so I won't try to give you advice on what a statistically significant result is. However many /many/ articles have been written on that topic, and many products have been designed which you can slot your data into to see if the 'B' option is legitimately better.

There are no comments on this comment yet.