News for hackers. Stuff that matters.
Re:Most Winning A/B Test Results are Illusory [pdf] by RyJones on Sunday February 23, 2014 @06:57PM (#7288312)
When ExP was a thing at Microsoft, we always ran an A/A test before we did experiments. We'd also do an A/A/B test to make sure the actual experiments were working.

http://www.exp-platform.com/Pages/default.aspx

Re:Most Winning A/B Test Results are Illusory [pdf] by darkmighty on Sunday February 23, 2014 @08:15PM (#7288566)
There are also some complex problems with assumptions that are infrequently addressed, e.g. maybe if a regular user sees a structural/cosmetic change he is more likely to look at that and click it, while that would fade away in steady-state.
Re:Most Winning A/B Test Results are Illusory [pdf] by RyJones on Sunday February 23, 2014 @10:17PM (#7288915)
True. We did everything we could to account for that, though. Someone chooses to clear cookies every time they load a page - not much we could do.