Re:Looks vs. Results: My ugly ad got 150% more clicks than a...
by akiselev on Sunday January 03, 2021 @06:03PM
(#25626121)
> The reason it feels intuitively broken is because the conversion rate is so low. But there were about 10k impressions in his test.
Can't be 95% confident in something if natural variability drowns out the effect. That kind of conversion rate is one bot RNG away from returning to the mean (noise), it just takes some domain knowledge to recognize why p-hacking is an illusion.
Re:Looks vs. Results: My ugly ad got 150% more clicks than a...
by canjobear on Sunday January 03, 2021 @05:51PM
(#25626015)
That’s not what significance means.
Re:Looks vs. Results: My ugly ad got 150% more clicks than a...
by NationalPark on Sunday January 03, 2021 @05:57PM
(#25626071)
I mean... yes it is? 95% confidence in an effect is what most people mean when they say "statistically significant".
Re:Looks vs. Results: My ugly ad got 150% more clicks than a...
by jonex on Sunday January 03, 2021 @07:04PM
(#25626531)
I think the objection is about the phrasing that can be interpreted as "I will only be wrong in at most 5% of significant results", while what we mean is "if there is no effect, we will only get so large differences in at most 5% of cases".
It's not uncommon that people fail to notice the nuance here and it's quite important for correctly interpreting reults.
The reason it feels intuitively broken is because the conversion rate is so low. But there were about 10k impressions in his test.