News for hackers. Stuff that matters.
Re:Looks vs. Results: My ugly ad got 150% more clicks than a... by contravariant on Sunday January 03, 2021 @05:20PM (#25625778)
Well, with low probability events you can go a bit further with your back of the envelop calculations, because that means they're more or less Poisson distributed. The average is somewhere around 16 so that gives a standard deviation of 4.

So there's about 3 standard deviations between the two, this sounds like quite a bit but really means they're both 1.5 standard deviations from the supposed mean. Which is, not great, though it might pass some of the weaker statistical tests.

Now you should actually weight the values by the total number of impressions in which case you might get a slightly higher significance since the one with 12 clicks was seen by more people.

So on the balance you should be wondering what you're paying the graphic designer for, but perhaps not start a new career designing low-budget adds.

Re:Looks vs. Results: My ugly ad got 150% more clicks than a... by whimsicalism on Sunday January 03, 2021 @05:43PM (#25625938)
Fisher's exact test p-value of 0.007 is pretty decent, not "not great."
Re:Looks vs. Results: My ugly ad got 150% more clicks than a... by contravariant on Sunday January 03, 2021 @06:27PM (#25626307)
I assume that one takes the uneven sampling into account? That should boost the significance a bit, though perhaps a bit more than I'd originally assumed. It remains a back of the envelop calculation after all.