News for hackers. Stuff that matters.
Re:Most Winning A/B Test Results are Illusory [pdf] by patio11 on Monday February 24, 2014 @04:44AM (#7289912)
under the assumption that week-week changes in normalized stats would be swamped by my tests

This is an enormously problematic assumption, which you can verify by either looking at the week-to-week stats for a period prior to you joining the company, or (for a far more fun demonstration) doing historical testing of the brand of toothpaste you use for the next 6 weeks. Swap from Colgate to $NAME_ANOTHER_BRAND, note the improvement, conclude that website visitors pay an awful lot of attention to the webmaster's toothpaste choices.

There are no comments on this comment yet.