So the effect could be tiny.
Worse, the report says it excludes observations outside the 0.5th and 99.5th percentiles of the 2021–22 fall-to-spring score-change distribution. I'm not sure how this is defensible. Either you trust your measurement instrument or you don't.
If you're using a difference-in-differences design, excluding observations can change what you're measuring, particularly if the exclusions differ between treatment and control groups or over time. Without trimming those observations, the estimated effect drops from 0.0087 SD to 0.0058 SD (and is no longer statistically significant).
The authors preregistered their study, but there's nothing in the report indicating that the trimming of 'extreme' values was part of that preregistration...
Wow! I love this.
I love that you bothered to email them with a thought out question.
I love that they bothered to respond with a thought out answer and they took you quite seriously.
I love that you bothered to tell us about the exchange.
Thank you so much for all of this. This is what we should all strive to do with our time and as a community.
i don't know why you give them the benefit of doubt and say it's "nothing to sneeze at." it's utterly meaningless. if it were the SAT, this effect size can be achieved by studying for 10 minutes instead of 0.
here's the right takeaway:
"people should be outraged that school systems are blaming phones, which conclusively have absolutely nothing to do with school performance"
- Trust the anecdotes and this effect can't be fully captured by the study - Trust the study and assume the parents are projecting their desire for banning phones to be a good thing.
I'm leaning towards #2, since I also have a strong desire to ban phones around my kids (they're 3 and 1(x2) so yes, around them not for them). I'd personally be really prone to confirmation bias myself around phone bans.
-Jeff Bezos, paraphrased