News for hackers. Stuff that matters.
Re:Most Winning A/B Test Results are Illusory [pdf] by ep103 on Sunday February 23, 2014 @05:20PM (#7287871)
How do you calculate the correct sample size for a test, to achieve the correct "power"?
Re:Most Winning A/B Test Results are Illusory [pdf] by gwern on Sunday February 23, 2014 @07:05PM (#7288336)
For binomial scenarios like a stock A/B test, most statistical environments will have some sort of built-in power functions. For example, R does; an example: http://www.gwern.net/AB%20testing#power-analysis
Re:Most Winning A/B Test Results are Illusory [pdf] by tel on Sunday February 23, 2014 @08:50PM (#7288673)
For simple tests you can reverse the mathematics to get good estimates of how many observations are needed given a goal for your desired power and tolerance for false positives. Asking for greater power makes your test more sensitive ("buys a bigger telescope") at the cost of increased sample size. Asking for fewer false positives ("cleaning the lenses") costs similarly.

For more sophisticated tests, ones less likely to be seen in an A/B scenario, you might not be able to reverse the mathematics and get a direct answer, so often people will run simulation studies to guess at the needed sample size.

Re:Most Winning A/B Test Results are Illusory [pdf] by darkxanthos on Monday February 24, 2014 @07:48AM (#7290464)
I use simulations so I can avoid math and analyze the power of any distribution I can model in a consistent manner.

I wrote something about it here: http://www.databozo.com/2013/10/12/Finding_a_sample_size_for...

EDIT: typo

Re:Most Winning A/B Test Results are Illusory [pdf] by carbocation on Sunday February 23, 2014 @05:42PM (#7288008)
Here are a few resources: http://statpages.org/#Power
Re:Most Winning A/B Test Results are Illusory [pdf] by jamiequint on Monday February 24, 2014 @03:57AM (#7289773)
Lehr's Equation (Rule of 16) is generally a good estimator. It is explained (but not referenced) in a similar article here: http://www.evanmiller.org/how-not-to-run-an-ab-test.html