> if VAM is measuring the amount of value a teacher adds to student test performance based on an average student of similar background, and the average student in a wealthy neighborhood does well regardless of the teacher, then every teacher in every wealthy neighborhood adds no value!

I'm no statistician, but I don't think this is actually true. We would expect students who are likely to do well on standardized tests under a Hypothetical Average Teacher (HAT) to benefit relatively less from a good teacher than students who are likely to do poorly on standardized tests under a HAT - at least when we use standardized test results as the metric. But presumably, any sophisticated statistical model will take this into account. You would simply adjust the model such than a unit increase in actual score vs. predicted score is "worth more" in terms of indicated teacher performance as predicted score increases. It's just a case of diminishing marginal returns, and not unique to teaching or education.

It might be harder to distinguish signal from noise as the predicted test score increases. But it still seems a better system than "the students have done well, therefore the teacher has done well".