How Optimizely Almost Got Me Fired (2014)

Re:How Optimizely Almost Got Me Fired (2014) by erikbern on Saturday January 09, 2016 @05:24PM (#10873139)

> Statistical power is simply the likelihood that the difference you’ve detected during your experiment actually reflects a difference in the real world.

This seems incorrect to me. Isn't statistical power the likelihood that the null hypothesis would generate an outcome at least as extreme as what you observed?

I'm guessing the issue has a lot more to do with peeking at the outcome and not correcting for it (and similarly running many tests)

http://www.stat.columbia.edu/~gelman/research/unpublished/p_...

[ Parent | on HN ]

Re:How Optimizely Almost Got Me Fired (2014) by smu3l on Saturday January 09, 2016 @05:36PM (#10873170)

Yes, this is incorrect.

Also the next sentence is wrong:

>You’ll often see statistical power conveyed as P90 or 90%. In other words, if there’s a 90% chance A is better than B, there’s a 10% chance B is better than A and you’ll actually get worse results.

Having a test w/ 90% power means that if A is truly better than B (for one sided) or A is truly different than B (two sided), then you'll detect it 90% of the time you run the test (on independent data).

[ Parent | on HN ]

Re:How Optimizely Almost Got Me Fired (2014) by brockf on Sunday January 10, 2016 @02:28AM (#10874566)

This post is almost entirely inaccurate re: actual statistics. Power, as you say, is the likelihood of a given experiment rejecting the null hypothesis given some a priori sample size and effect size. And one- vs. two-tailed tests do not change effect sizes, or even your estimate of the variability surrounding an effect, but a p-value related to the effect (should you choose to calculate one).

You would think, given their team of "analysts" and "statisticians", that they might have known these basic pieces of statistics.

[ Parent | on HN ]