Thus the results are not statistically significant aka with in normal random error etc. So when did 0.3 of not significant become significant.
From the wikipedia article on statistical significance (http://en.wikipedia.org/wiki/Statistical_significance): "Given a sufficiently large sample, extremely small and non-notable differences can be found to be statistically significant, and statistical significance says nothing about the practical significance of a difference."
This isn't even "extremely small". From the report: "The mean treatment effect of 0.22 SD is equal to 9 percentile points at the median of a normal distribution. We find a minimum average treatment effect of 0.1 SD at every percentile of baseline test scores, suggesting broad-based gains in test scores as a result of the incentive program."
"At the end of two years of the program, students in incentive schools performed significantly better than those in comparison schools by 0.28 and 0.16 standard deviations (SD) in math and language tests respectively...."
That's 0.28 for math and 0.16 for language. I'd say that's pretty significant - especially for a large sample size. In this case presumably because of the sample size it's more than being about "power/tools/money".