It should be per software. Maybe per software version, if the company can conclusively prove that the prior problems that led to points cannot happen again.

How can you prove that without putting them back out on the road first? Surely, you'd agree that more is needed than just unit testing?

Closed course real world testing. I naively expect that you could test most of the listed situations in a small-medium size parking lot.

A driving exam also happens on the road. I'd suggest a similar, supervised test series with mandatory emergency drivers.

Do points get deducted or erased when the AI upgrades version? If the AI uses semver, how many points off for a major revision release?

Per driving license. It already works like that with truck driver and other professional drivers. They may have 5x more accidents than normal person, but they drive 20x more miles.

Truck drivers are individual people making discrete decisions. A robocar company all uses the same training data to make the same results.

Put 10 different human drivers in the same situation, you'd expect a variance of decisions being made. Put 10 robocars in the same situation, you'd expect 1 decision being made. Is this not how they are being tested? Am I crazy to assume this?

Absolutely, however holding the entire fleet responsible for the number of faults that would disqualify a single driver also seems wrong. If a taxi driver drives 8 hours a day, then 10 robotaxis will drive 80 hours in that same day and statistically be subject to perform 10x the number of faults, if we're holding them to the same standard.

Now, if 12 points loses your license, you have 10 cars, and rack up 120 points across the fleet, it seems obvious that on the whole the fleet is performing worse than a fleet of human drivers we would otherwise ban.

Not an easy problem, to be sure. On one hand, companies can't be allowed free reign to clog up the roads and cause havoc without consequence. On the other, if we NIMBY the development too much, we will be stuck behind countries that don't. In my opinion, we're near the right balance but probably need to take some short term action to make it clear to companies that a remedy for the current behavior around emergency vehicles needs to be priority 0.

If you take 10 exact same calculators and perform the same operation, you're going to get the same result on all 10. Why? It's a program running on the same hardware. If this hypothetical fleet of robocars are all on the same hardware using the same software, they are essentially the same machine. This one of the key things we've been told about how much safer robocars will be compared to humans. These robocars are not sentient making unique decisions. They all have the same set of logic in them. Your logic just does not compute with me.

Edit: also, it's the same as a recall to me. If one part is deployed to thousands of cars, all of them are recalled. Same applies here for me.