News for hackers. Stuff that matters.
Re:The Men Who Stare At Screens (why going to the gym might ... by _delirium on Thursday July 15, 2010 @12:48PM (#1518319)
I could be wrong, but at least in my little corner of science, this particular kind of post-hoc analysis, where you're moving thresholds on a linear scale for reporting purposes, seems not to be viewed with great suspicion, because it doesn't really produce arbitrary degrees of freedom. The worse kind of post-hoc analysis is where you can end up claiming things like "men who watched TV between 5.5 and 9 hours a week [whatever] more than men who watched TV either 1.0-2.0 or 11.5-12.5 hours/week", retroactively discovering a million probably random correlations in your data. Though even there there are ways of reducing the risk of doing so; for example, cross-validation is commonly used to test whether discovered correlations are artifacts of sampling or not, and has some pretty solid statistical theory behind it.

But in this case the general hypothesis is pre-chosen and fixed, basically "men who watch TV more hours [whatever] more than men who watch TV fewer hours", and the choice of threshholds is more of a reporting step, pulling out some binned tails at either extreme for emphasis and ease of discussion, while the real correlation is full-curve-to-full-curve.

Re:The Men Who Stare At Screens (why going to the gym might ... by carbocation on Thursday July 15, 2010 @01:34PM (#1518468)
I largely agree with you. I view moving thresholds with suspicion when you're trying to report effect sizes, but sure, I mean, I don't doubt the association. I doubt the magnitude, not the association per se.

And I absolutely agree that a sort of nonparametric post hoc approach that you poo-pooed would be even more highly suspect. But I don't see the choice of thresholds as simply a reporting step that can be manipulated ad lib without adjustment of your belief in the association and magnitude of effect. (I don't mean to imply that you think that you can adjust cutoffs willy-nilly - just stating my opinion clearly here.)