Re:Does Your Product Logo Actually Matter?
by patio11 on Wednesday August 04, 2010 @03:06PM
(#1575471)
Let's not talk A/B tests and talk medicine for a moment, shall we? Here we have aspirin, here we have a placebo. There isn't a population of aspirin pills and a population of placebo pills. There aren't many different populations of placebo pills. The populations are of people. We're trying to make inferences about the population of people exposed to aspirin versus the population of people exposed to a sugar pill, but since it is not possible to bring all of them in, we make those inferences based on random samples of those two populations.
Now here is where I start getting murky: results for the particular things we're interested in are binary: the aspirin worked or it didn't. You converted or you didn't. The Central Limit Theorem says that, if you sample n individual unconnected random binary events from a population, and you keep repeating that sample, the distribution of your measured percentage of success will approximate the normal distribution, centered around (with the mean on) the true population statistic.
That is important for a couple of reasons, but the most immediate one is that it gives us a statistical figleaf to hide under for using two samples to make inferences about whether the population mean for the first distribution is highly likely to not be the same as the population mean for the second distribution. i.e. whether the A and the B are likely to be materially different.
So let's walk this back to where we were talking about potentially testing multiple different types of sugar pill. Do you get why that is sort of incoherent in this framework? There are key assumptions like "independence" that two different types sugar pills surely do not share, but I don't know if you can even get to that objection before you deal with the notion that there is no coherently cognizable population made out of sugar pills.
It is 4 AM in the morning. I am not sure that is stats class talking and not sleep deprivation. This is probably covered in a good Stats 101 book -- if you have further questions, they're likely more coherent than I am. (Or ask btilly -- he has a gift for explaining this.)
Now here is where I start getting murky: results for the particular things we're interested in are binary: the aspirin worked or it didn't. You converted or you didn't. The Central Limit Theorem says that, if you sample n individual unconnected random binary events from a population, and you keep repeating that sample, the distribution of your measured percentage of success will approximate the normal distribution, centered around (with the mean on) the true population statistic.
That is important for a couple of reasons, but the most immediate one is that it gives us a statistical figleaf to hide under for using two samples to make inferences about whether the population mean for the first distribution is highly likely to not be the same as the population mean for the second distribution. i.e. whether the A and the B are likely to be materially different.
So let's walk this back to where we were talking about potentially testing multiple different types of sugar pill. Do you get why that is sort of incoherent in this framework? There are key assumptions like "independence" that two different types sugar pills surely do not share, but I don't know if you can even get to that objection before you deal with the notion that there is no coherently cognizable population made out of sugar pills.
It is 4 AM in the morning. I am not sure that is stats class talking and not sleep deprivation. This is probably covered in a good Stats 101 book -- if you have further questions, they're likely more coherent than I am. (Or ask btilly -- he has a gift for explaining this.)