Re:Don't calculate p-values without understanding them
by glaugh on Thursday May 09, 2013 @02:05PM
(#5681969)
Yeah, my point isn't that the statistical significance is erroneous or that the difference shouldn't have been significant.
In my totally made up example, there's really a .1mm difference. And maybe to someone that's meaningful, so it's worth reporting. But it's not a big enough effect to say "When I see someone on the street I can look at their height and know which of these two cities they're from."
And my peeve is that people will see a statistically significant difference and think that that alone makes it an important finding, versus a very real finding that might not actually matter that much.
Or, related, it's bad to see something with a p-value of .15 and a large effect size and decide that since it's not significant at .05 it's not interesting. Since that combination typically means that you have a too-small sample size, the best interpretation is probably more like "This might be interesting. Looks like we don't have enough data to tell if there's a relationship here, but it probably deserves another look."
(Edited to add that last thought, and clarify the previous one)
In my totally made up example, there's really a .1mm difference. And maybe to someone that's meaningful, so it's worth reporting. But it's not a big enough effect to say "When I see someone on the street I can look at their height and know which of these two cities they're from."
And my peeve is that people will see a statistically significant difference and think that that alone makes it an important finding, versus a very real finding that might not actually matter that much.
Or, related, it's bad to see something with a p-value of .15 and a large effect size and decide that since it's not significant at .05 it's not interesting. Since that combination typically means that you have a too-small sample size, the best interpretation is probably more like "This might be interesting. Looks like we don't have enough data to tell if there's a relationship here, but it probably deserves another look."
(Edited to add that last thought, and clarify the previous one)