News for hackers. Stuff that matters.
Re:How to share data with a statistician by sanskritabelt on Wednesday November 27, 2013 @02:08AM (#6806905)
Maybe I was unclear. To be explicit, the experimental setup was: * Draw a sample of objects (in this case, an 'object' was a set of text documents) * Apply the OLD and NEW treatments to each sampled object * Show the results of OLD and NEW to volunteers (double blind, of course), get an evaluation.

And so the data I get is: (object 1, SCORE OLD 1, SCORE NEW 1) (object 2, SCORE OLD 2, SCORE NEW 2) etc

So again: * I don't have any prior knowledge of how the scores are going to be distributed * The primary reason I expected OLD to be 'better' than NEW is because OLD had theory behind it, and NEW was an ad hoc tweak of a previously rejected treatment * The very initial text corpus was fixed * The objects were selected from the initial corpus through a multistep process, at least one of those steps was a stochastic optimization of an objective function * One of the treatments was also obtained through a constrained optimization * The initial text corpus wasn't all that large, and the samples were pretty small

So what I did was look at all observations for which OLD SCORE was different than NEW SCORE. Say there were n of them, and say that for p of those OLD<NEW. If NEW wasn't producing results that scored better than OLD, we would expect p/n to be less than 0.5, up to some uncertainty from the random sample. This is just a straight up test of a binomial parameter, so I computed confidence intervals and found that 0.5 was several standard deviations below the observed p/n, from which I concluded that NEW was better than OLD and we should switch. (as described here: http://en.wikipedia.org/wiki/Sign_test)

In a more responsible analysis I could have done something more sophisticated and modeled the scores as a sum of object effect, tester effect, treatment effect, fixed term, noise, etc. Given the magnitude of the sign test score, I didn't feel that was necessary.

Re:How to share data with a statistician by loup-vaillant on Wednesday November 27, 2013 @05:06AM (#6807370)
So, there was no "first source of uncertainty", and you had the advantage of real-world observation and photos put together. Neat.

Indeed, given the apparent massive evidence in favour of NEW, you probably didn't need to twist your brain further.

Now, if we're doing A/B testing on a commercial web site, I think we should use every scrap of information available to us, and go into full Bayesian mode if at all possible: if of A and B, one is worse, you don't want to make one too many test before you figure this out. The sooner you positively know which is best, the better.