Your quote concerns variance, not bias.
(The main exception is the bit about "full range of achievement". It's quite true that teachers of upper class Asian students will likely have lower VAM scores than they deserve since the test caps out at 100%.)
We already knew that the contributions of teachers to student achievement was relatively small long before VAM was invented.
You dodged the question. Do you think policymakers and politicians should stop trying to improve teachers, and instead simply try to reduce costs? I.e., pay teachers less, increase their turnover (so that we don't have to pay expensive but ineffective senior ones), etc?
What were you originally referencing in terms of bias?
> Do you think policymakers and politicians should stop trying to improve teachers, and instead simply try to reduce costs? I.e., pay teachers less, increase their turnover (so that we don't have to pay expensive but ineffective senior ones), etc?
I mean the answer to any of those types of questions really depend on what kind of school system you want to create as a whole, you could go either way on any of them and have either a good or a bad school system depending on the overall design choices. But my point was that I don't think that VAM has much to say about any of these questions, because whether or not it's theoretically possible to measure teacher performance doesn't say much about whether or not good and bad teachers actually exist and whether or not they have an impact on students learning/lives. My point was that the reason we can't measure teacher performance isn't because the difference between good and bad teachers is small, even though it is relatively small in the grand scheme of things.
http://en.wikipedia.org/wiki/Variance
Alex, I'd suggest you refrain from opining that standard statistical methods fail in education until after you master basic statistics.
E.g. if the scores have no stability so who gets fired is basically just a coin toss (as these links suggest) then that's really important, but if these tests systematically overestimate or underestimate teacher performance then that doesn't really matter.
"As expected, the level of uncertainty is higher when only one year of test results are used [...] as against three years of data [...]. But in both cases, the average range of value-added estimates is very wide. For example, for all teachers of math, and using all years of available data, which provides the most precise measures possible, the average confidence interval width is about 34 points (i.e., from the 46th to 80th percentile). When looking at only one year of math results, the average width increases to 61 percentile points. That is to say, the average teacher had a range of value-added estimates that might extend from, for example, the 30th to the 91st percentile. The average level of uncertainty is higher still in ELA [English Language Arts]. For all teachers and years, the average confidence interval width is 44 points. With one year of data, this rises to 66 points."
"The Mathematica models, which apply to teachers in the upper elementary grades, are based on two standard approaches to value-added modeling, with the key elements of each calibrated with data on typical test score gains, class sizes, and the number of teachers in a typical school or district. Specifically, the authors find that if the goal is to distinguish relatively high or relatively low performing teachers from those with average performance within a district, the error rate is about 26% when three years of data are used for each teacher. This means that in a typical performance measurement system, more than one in four teachers who are in fact teachers of average quality would be misclassified as either outstanding or poor teachers, and more than one in four teachers who should be singled out for special treatment would be misclassified as teachers of average quality. If only one year of data is available, the error rate increases to 36%. To reduce it to 12% would require 10 years of data for each teacher."
So even with three years of data, the scores are fairly unstable, and this isn't even taking into account the fact that even if they are stable they aren't necessarily correct -- there are many more sources of error that would still be stable over time.
Furthermore, short term student gains correlate poorly with longterm student gains, so if you wanted to measure longterm student gains (presumably the most important metric) then you'd have to wait an addition couple of years after each grade before testing. This means that, in addition to the 3+ years of data you'd need to have even a semi-stable variance, it would take at least 5 years total to have even decent data:
Economist Jesse Rothstein found that students' learning decays over time, and that the teachers who had the biggest added value gains at the end of the first year didn't necessarily have the biggest gains at the end of years two or three when the same students were retested. According to the Rothstein paper, "Three aspects of the results are of note. First, there is much more variation in fourth grade teachers' effects on fourth grade scores than in those same teachers' effects on fifth grade scores. [...] Second, the average persistence of fourth grade teachers' effects one year later is only around 0.3, again consistent with recent evidence. Third, the data are not even approximately consistent with the notion that this persistence rate is uniform across teachers: The correlation between teachers' first-year effects and their two year cumulative effects is much less than one, ranging between .33 and .51 depending on the model and subject. Three-year cumulative effects show a similar pattern, correlated around .4 with the immediate effect. Even if we assume that the VAM-based estimates can be treated as causal, a teacher's first-year effect is a poor proxy for his or her longer-run impact. [...] Only about a third of teachers in the top quintile of the distribution of two year cumulative effects are also in the top quintile of the one-year effect distribution. It is apparent that misspecification of the outcome variable produces extreme amounts of misclassification. [...] A full account of the utility of VAMs for identifying good teachers would need to combine the analyses of lagged effects and endogenous classroom assignments. This would imply even higher rates of misclassification than are produced by either on its own."
Sounds to me like "teachers don't matter, lets try to cut costs". Do you disagree with this?
"VAM’s instability can result from differences in the characteristics of students assigned to particular teachers in a particular year, from small samples of students (made even less representative in schools serving disadvantaged students by high rates of student mobility), from other influences on student learning both inside and outside school, and from tests that are poorly lined up with the curriculum teachers are expected to cover, or that do not measure the full range of achievement of students in the class."
> If you believe VAM is unable to measure the effects of teachers, are you willing to follow through with the logical conclusions? Namely, the effect teachers have on students is so small it can't be measured ...
We already knew that the contributions of teachers to student achievement was relatively small long before VAM was invented. The only people who seem to think that teachers have a very large impact are the pro-VAM folks (e.g. Bill Gates and Malcolm Gladwell), who all cite the same flawed study.
That said, the reasons why VAM can't measure teacher performance isn't because the contributions of teachers are small per se, but rather because there are all sorts of problems with all of the proposed measurement systems.
In order to create a VAM system that actually worked, assuming such a thing is possible, you'd have to make a number of changes to the way that the school system is designed... e.g. assigning students to teachers at random, designing standardized tests to measure teacher quality, implementing standardized tests in all subject areas, etc. But implementing all of those changes may well be worse for student learning than firing all the bad teachers or whatever. Not to mention that there is currently a major crisis in teacher retention, with half of all teachers either getting fired or quitting in their first five years. (After spending two years and god knows how much money getting their masters degrees to go into the profession.)