News for hackers. Stuff that matters.
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by yorwba on Tuesday November 17, 2026 @08:38AM (#50531186)
The study https://cepr.harvard.edu/sites/g/files/omnuum9881/files/2026... finds a ≈0.01 standard-deviation difference in test scores the year lockable phone pouches are adopted, with no detectable additional difference in the following year. Certainly nothing to sneeze at from a system-wide perspective, but easily overwhelmed by other factors for any individual student.
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by contravariant on Tuesday November 17, 2026 @08:57AM (#50531447)
Individual test students are going to be hard to predict anyway. Though I think this is compared to the sample standard deviation, and for most students their improvement over time is going to be smaller than the difference between two random students.
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by rahimnathwani on Tuesday November 17, 2026 @09:28AM (#50531865)
That ≈0.01 SD effect is actually 0.0087 +/- 0.0043 (standard error), giving a 95% confidence interval of roughly 0.0003 to 0.0171 standard deviations.

So the effect could be tiny.

Worse, the report says it excludes observations outside the 0.5th and 99.5th percentiles of the 2021–22 fall-to-spring score-change distribution. I'm not sure how this is defensible. Either you trust your measurement instrument or you don't.

If you're using a difference-in-differences design, excluding observations can change what you're measuring, particularly if the exclusions differ between treatment and control groups or over time. Without trimming those observations, the estimated effect drops from 0.0087 SD to 0.0058 SD (and is no longer statistically significant).

The authors preregistered their study, but there's nothing in the report indicating that the trimming of 'extreme' values was part of that preregistration...

Re:New Evidence Shows School Phone Bans Are Gradually Improv... by oofbey on Tuesday November 17, 2026 @10:17AM (#50532522)
Sounds like they noticed a few extreme outliers were pulling the difference down and hacked it out. Methodologically that’s really sus to do this after the study is done. But generally I think it’s okay to exclude extreme outliers - we generally assume Gaussian distributions, but the real world happens and messes things up. Having a few kids that are 5 or 6 sigma out pulling your averages - I can believe the rules are different for them. But this should have been decided ahead of time.
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by rahimnathwani on Tuesday November 17, 2026 @10:21AM (#50532588)
Just to be clear, I can't say for sure that the preregistration didn't include the extreme value exclusion. I haven't been able to find a link, but I will email the authors to ask for it.
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by Balgair on Tuesday November 17, 2026 @11:31AM (#50533869)
Aside:

Wow! I love this.

I love that you bothered to email them with a thought out question.

I love that they bothered to respond with a thought out answer and they took you quite seriously.

I love that you bothered to tell us about the exchange.

Thank you so much for all of this. This is what we should all strive to do with our time and as a community.

Re:New Evidence Shows School Phone Bans Are Gradually Improv... by dryanau on Tuesday November 17, 2026 @02:00PM (#50536447)
Alright mate
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by buellerbueller on Tuesday November 17, 2026 @02:37PM (#50536982)
Yeah! Fuck follow-through!
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by oofbey on Tuesday November 17, 2026 @12:26PM (#50534897)
Interesting that their take is that the outliers represent measurement errors. Plausible and reasonable. But the fact that it changes the conclusion of the study deserves skepticism.
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by gamblor956 on Tuesday November 17, 2026 @07:10PM (#50540124)
The outliers would have made the claimed effect even stronger. They were trying not to overhype their findings.
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by rahimnathwani on Wednesday November 18, 2026 @12:14AM (#50542142)
This is incorrect. Please see my earlier comment.
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by rpdillon on Tuesday November 17, 2026 @01:15PM (#50535725)
Phenomenal. This is why I come to Hacker News. Thanks so much.
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by piker on Tuesday November 17, 2026 @03:32PM (#50537761)
Respect! Good science
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by doctorpangloss on Tuesday November 17, 2026 @09:55AM (#50532224)
is this really the study the nytimes references?

i don't know why you give them the benefit of doubt and say it's "nothing to sneeze at." it's utterly meaningless. if it were the SAT, this effect size can be achieved by studying for 10 minutes instead of 0.

here's the right takeaway:

"people should be outraged that school systems are blaming phones, which conclusively have absolutely nothing to do with school performance"

Re:New Evidence Shows School Phone Bans Are Gradually Improv... by tgv on Tuesday November 17, 2026 @10:33AM (#50532782)
Don't forget that scores are normalized for many tests. It's not a linear scale from "nothing correct is a zero, everything correct is a 10". Rather, the average number of correct answers (often weighted, but ok) is a pass. This will mess up any analysis.
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by yorwba on Tuesday November 17, 2026 @11:18AM (#50533638)
The test they're using is the NWEA MAP Growth https://www.nwea.org/map-growth/ which doesn't have a concept of pass or fail. It doesn't even have a fixed set of questions, but adapts the difficulty dynamically based on earlier answers to get a more accurate measurement.
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by odo1242 on Tuesday November 17, 2026 @12:46PM (#50535231)
Yea, this study seems pretty bad
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by seizethecheese on Tuesday November 17, 2026 @02:30PM (#50536880)
Contrast that with all the anecdotes in this thread from parents about how great banning phones is, I'm left with two possible conclusions:

- Trust the anecdotes and this effect can't be fully captured by the study - Trust the study and assume the parents are projecting their desire for banning phones to be a good thing.

I'm leaning towards #2, since I also have a strong desire to ban phones around my kids (they're 3 and 1(x2) so yes, around them not for them). I'd personally be really prone to confirmation bias myself around phone bans.

Re:New Evidence Shows School Phone Bans Are Gradually Improv... by buellerbueller on Tuesday November 17, 2026 @02:35PM (#50536960)
When the anecdotes and the data point in different directions, trust the anecdotes; it usually means you are measuring the wrong thing.

-Jeff Bezos, paraphrased

Re:New Evidence Shows School Phone Bans Are Gradually Improv... by gruez on Tuesday November 17, 2026 @07:20PM (#50540205)
What does this mean for what is arguably the most famous data-anecdote gap: crime rates? People have been thinking crime has been skyrocketing for decades (well before the pandemic), but statistics show the opposite?
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by buellerbueller on Wednesday November 18, 2026 @10:21AM (#50547401)
Very simple: the news is not anecdotal; the news is editorial. If you were to talk to people about their own direct experiences with crime, I am confident that they would look far more like the crime stats. Instead, you have in news media an industry that chooses what to cover, and it never covers: "this part of town had no crime today," which is an anecdote itself.
Re:New Evidence Shows School Phone Bans Are Gradually Improv... by doctorpangloss on Tuesday November 17, 2026 @04:38PM (#50538552)
the tests they are using are measuring something. they just aren't measuring what everyone wants to use these tests for (admissions, explaining differences in race and gender, closing schools, picking the best schools, how to assign scholarships / give kids IEPs, the impact of cellphone bans, etc.)