Re:Experiment: Paying for performance works (for teachers in...
by yummyfajitas on Wednesday September 30, 2009 @12:28PM
(#852864)
They also accounted for it by studying performance on science and social studies tests. Both improved. However, it would be useful to follow up with a differently structured test to see how much comes from test prep.
Incidentally, bringing up the SAT is a red herring. The SAT was originally meant to measure g/intelligence/"aptitude". It did this by measuring a body of knowledge which is not explicitly taught in school, giving questions like wheel : car -> pick one of [leg: horse, egg : chicken, computer : TV]. Obviously, you can improve performance by explicitly teaching that knowledge (e.g., memorizing analogies).
A subject test does not suffer from this problem, since it is designed to measure subject knowledge rather than aptitude. You can improve performance on a multiplication test by memorizing multiplication tables, but so what? Multiplication performance is what you are trying to measure, it's not a proxy for something else.
That is assuming the unlikely proposition that preparing for the test doesn't directly prepare you for the conceptual portion of the test. I say unlikely because test makers have been trying to make such tests (eg the SAT) for decades, and test preparation companies (eg Kaplan) have been demonstrating that you can prepare for them after all.
You can't escape the circularity. If you're measuring performance with a test, then you can't really distinguish preparation for the test from actual performance. In this study they had much stronger results in their second year, than their first. Which strongly suggests that teachers did a better job of preparing for the test after teachers saw the tests that would be used.