Then you can obviously calculate what the model outputs for any set of inputs. But that's not measuring; that's modeling. They're not the same thing. You can't measure the students' performance with a hypothetical "average" teacher because that teacher does not exist. You can only model what you think are the relevant factors involved and then input actual data into the model to see what it says.
Can such models be informative? Certainly. Are they a substitute for human judgment? No way. Yet that is what these schools appear to be doing.
> if the best teachers at schools full of poor kids could get more incentive pay by switching to schools full of rich kids, would that be incentivising the right thing?
Does that actually happen? And if it does, would this model prevent it? Once again, models are no substitute for human judgment.
The proposed model makes no assumptions about the "average" teacher -- it proposes a simple metric applied to student performance. Under this metric, some teachers would get a positive score because their students' scores improved more than the average. Others would get a negative score because their students' scores improved less than the average.
> would this model prevent it?
I don't know, but the goal of the model is to prevent this, or even reverse it -- that is, the best teachers would want to work with the least advantaged kids, because arbitrage -- it should be easier to make a difference for kids starting at a low performance level, and harder to make a similar difference with kids starting at a higher level.
I don't know whether this works even a little bit in practice, but the thinking sounds promising.
Sure it does. How else is the teacher_skill rating calibrated?
> the best teachers would want to work with the least advantaged kids
Why is this necessarily a good thing? Shouldn't there also be an incentive to have the best teachers work with the brightest students, to ensure that those students are actually challenged instead of just skating through school?
And you're really left with c*teacher_skill + noise, where the noise in the teacher_skill measure may be what results in some good teachers getting bad ratings.
In my view, empirical modeling assumes that there is some underlying regularity that can be modeled, but this assumption seems questionable. And part of the process has to be to decide an acceptable level of good teachers getting bad ratings.
score_at_end_of_year = score_at_start_of_year + a * parents_income + b * parents_education_level + c * teacher_skill + random_noise
You can figure out the values of a and b by comparing the performance of different students in the same class. If you have a large enough sample size to average out the random noise, you're left with (c * teacher_skill).
Not a perfect method, obviously - but if the best teachers at schools full of poor kids could get more incentive pay by switching to schools full of rich kids, would that be incentivising the right thing?