News for hackers. Stuff that matters.
Re:Most Winning A/B Test Results are Illusory [pdf] by joosters on Sunday February 23, 2014 @06:40PM (#7288256)
http://www.evanmiller.org/how-not-to-run-an-ab-test.html gives a good explanation about how this can ruin your tests.
Re:Most Winning A/B Test Results are Illusory [pdf] by darkxanthos on Wednesday February 26, 2014 @09:56PM (#7310040)
He even says this is possible via Bayesian experiment design:

FTA: Bayesian experiment design: With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences.