Sort of the same way that companies do it, really. Engineering firms, or good ones anyway, rely more on subjective judgment of management ("so-and-so is a [terrible|great] engineer") than they do on code metrics. I think they'd have some of the same morale problems if they were heavily metrics-based as well: imagine if every year your company ran a script over their VCS+bug-tracker with whatever metrics were currently in place, and fired everyone who scored in the bottom 5%.

Also, have you ever written software for the government? I think if you had, you'd yearn for an objective evaluation procedure as accurate as standardized tests. For your proposal to work, you'd have to carefully set up incentives for the judges and try to figure out how to keep the politics to a minimum.

If "teaching to the test" is a bad thing, that just means you have a bad test. With varied enough questions, the easy way to prepare for a test is to understand the material. I agree that teachers shouldn't be judged directly by the scores of their students, because not all students have equal background and ability. But any time you have an opportunity to measure something objectively, jump at the chance. That goes for software, too.

You can judge most of the goals in retrospect, say 20 years out: what proportion of kids ended up in jail, what proportion ended up with good jobs, what proportion are informed citizens capable of reading the news and participating intelligently in the political process, what proportion are able to handle their personal finances, nutrition, health-care, etc. successfully, what proportion won prestigious awards or started successful businesses, and so on. But writing a test for 18-year-olds to take that predicts all that accurately is pretty hard, and as far as I know, no tests have shown even plausible evidence of their predictive power in that respect, especially when you control for demographics.

Look at it this way: We do not want students to all know the same things. We want to increase the amount of learning, but ideally, only 30-50% of the knowledge kids learn should be held in common. In a perfect world, the remaining 70% would be different from classroom to classroom. Varied experience is crucial to making our society function better.

So if we take varied experience as a value we want to encourage in our classrooms, you cannot have a good standard test, because the standard is more amount and quality of knowledge than a predefined notion of what knowledge is necessary to get by in the world.

And likewise, because this is how I value it, I would prefer that standardized testing be around one-third of the evaluation score, if not 20%. I want teachers and school administrations to be free to take big risks like teaching everyone computer science instead of math - because teaching a well-thought-out course of study is not a risk. Classrooms should definitely do instruction in ideas of the past (history is always vital), but more than anything they should be incubators for new ideas.

Now, the current system focuses on meeting minimums, and most standardized tests quickly hit a ceiling. I'd much prefer to see tests that attempt to gage the entire breadth and depth of student learning. Practically, that's going to require different questions asked of different students, or you'll end up with a 10,000 question exam. But I still consider it standardized testing. The SAT and AP exams are great - we should do more of that earlier.

Sure they are, but to what degree do they really reflect on the teacher? A teacher can be doing a fantastic, patient job with a student, and make zero progress all year with a student. That work is meaningful, but its effect could take years to show up. Can we measure this? Sure. But it will be very error-prone, and ultimately less effective than just asking the other people in the building who is worthless (everyone knows except the worthless themselves.)

We're not looking to find the best and brightest - we as a society have made the decision that we're not going to pay that much. We just want to get rid of the unequivocally awful.

There needs to be some attempt at normalization between teachers and school districts, of course. But there's room for qualitative measures instead of quantitative ones. Especially when the tests seem (unintentionally) designed to punish teachers for teaching poor/minority/ESL students.

Especially when the tests seem (unintentionally) designed to punish teachers for teaching poor/minority/ESL students.

This is utterly false if we reward teachers based on deltas rather than absolute values, that is we compare actual performance to statistically predicted performance and reward/penalize teachers when actual performance exceeds predicted performance.

Isn't there a DailyWTF about a company that did just that and found that the testing people conspired with the developers to invent bugs, locate them, and then collect bonuses for fixing them.

Why should they want to work under a system like that?

What independent agency would you want devising metrics to figure out how good you are at your job while knowing nothing about the specific challenges you deal with?

The independent agency which currently devises metrics to figure out how good I do my job is the stock market. My bonus = % of profits.

When I was a professor, I favored (and suggested to the department) that some higher authority should devise midterm and final exams. It would a) allow us to measure teacher performance, as opposed to student opinion [1] and b) allow comparison of student performance across classes.

[1] To get good student evaluations, all you need is easy tests + jokes + good looks.

Public education isn't a market good, it's a public good. Toilet paper and oatmeal are market goods.

The labor market for teaching is in fact a free market. Any given municipality can pay whatever they want to pay, and get the talent they're paying for accordingly. The fact that teachers are getting laid off in something like 80% of municipalities the last few years means you don't even have to scrape the bottom of the barrel.

Agree on the statements about professors. All the same. If you were working on something more qualitative as part of a team, and not a lone trader, we both know there's no quantitative way to assess your work. Lines of code is a joke and other metrics invariably bring in subjective measures.

EDIT to add: If you're pro-free market, shouldn't it be up to municipalities to choose which teachers they want to keep or fire as they see fit, without some bureaucrat telling them that according to some BS test scores they have to do such and such?

Also, the labor market for most govt employees is not remotely a free market. Unions keep out the competition, essentially getting no-bid contracts to supply labor. They play politics to elect politicians who will give them what they want - in many cities the teachers unions essentially control the political system. You might as well call Halliburton's no-bid contracts in Iraq a "free market". It's a "free market", closed to new entrants, with no negotiation between the buy and sell side, but with the buy and sell side colluding to rip off the public.

In any case, just because it's tough to quantitatively measure some development does not mean it's tough to measure teacher performance. Students either know the material at the end of the term or they don't, simple as that.

All the whining about how it's impossible is really just an attempt by teachers unions to avoid accountability. I'm sure Halliburton would love to avoid accountability for their performance in Iraq and elsewhere too, doesn't mean we should take them seriously.

In almost no city does the teacher's union essentially control the political system. The police union is always, always more powerful, for one. They're just more politically engaged, people who are driven by power. Several private unions will be more powerful. And that's leaving out every power broker that isn't a union (representing most of the power).

"All that whining about how impossible it is to measure programmer productivity is just an attempt by developers to avoid accountability." You might consider that you don't know a single thing about teaching in a non-college or lower-class environment. No offense. I don't claim to, either. But I'll at least listen to what every teacher I've ever met says about the matter.

So what's the analogous mechanism going to be? Are you ok closing entire schools if they have lower than average test scores in their class (whatever this means)? Otherwise, how do prevent the whole system from devolving into politics and cronyism?

Even if you are exceptional in this regard, the average person will not do well. And I have read a lot of reviews in my life from college students.

Until you know the subjects well, you are only evaluating whether or not you liked it.

That's a big if. From what I've seen, this is practically never done - absolute values are the norm.

As an aside, pointing out that something is bad does not put the onus on me to invent something better. The sensible conclusion is to not do anything until something good has been worked out. The alternative is to do something just for the sake of appearing to do something.