News for hackers. Stuff that matters.
Re:Results from Khan Academy experiment by john_horton on Friday September 02, 2011 @06:05AM (#2953560)
In the comments, the authors give the sample size as about 20 in each group. Here's a more sensible graph (error bars to create a 95% CI on the estimated group means):

https://skitch.com/johnjosephhorton/fih62/quartz-2

Obviously, there are some statistical power issues. It's unfortunate that they didn't get someone to help them with a power analysis up front---given how many very tech savvy people are interested in Kahn Academy and want to see it succeed, I think they could have gotten this sort of assistance pretty easily.

Re:Results from Khan Academy experiment by JunkDNA on Friday September 02, 2011 @08:59AM (#2953913)
Thanks for doing that, I found myself wishing they had put error bars on when I saw that graph but was too lazy to do it myself.

With respect to the statistical power issue, it may be they just didn't have enough students. Unlike your typical clinical trial or scientific study where you're recruiting people, I got the impression this was from an existing pool of students for a specific school. Once you go beyond this, the design gets tricky since there are practical issues to how big a class can be. Then you need more teachers, and you need to normalize for per-teacher variability. Not saying a study like that wouldn't be great, but it would be complex to get right.

Re:Results from Khan Academy experiment by john_horton on Friday September 02, 2011 @01:21PM (#2954826)
no problem & agreed about what probably happened - my point was more that knowing they had that sample size (n=40), someone could have told them that even if Kahn academy was extraordinarily effective, there was almost no chance they'd detect statistically significant effect. If they knew that, they might have pursued a more qualitative approach rather than have people see the take-away as "no effect."