How would you detect these changes in a licensed system if you are allowed to modify the vehicle? If you do behavior based testing - the person will just hide the features behind a backdoor so the testing looks normal. If you do some type of checksum based testing, you have defacto made modification illegal, and it is bypassed by doing a "factory reset" prior to the licensing check, and loading the modified software after the licensing check.

If you try to do some type of remote real-time detection of an unlicensed system, software that is modified can masquerade as unmodified.

What's to stop someone from driving without a license under the current system? A bunch of cops looking for illegal and unsafe behavior. If you're driving without a license but in a manner that cannot be detected then you're not likely be causing harm anyway, so letting you "slip through the cracks" is no great loss to society.

If someone modifies their car's software to tailgate and pass other vehicles at speed on the shoulder they'll likely be detained, investigated, and prosecuted. If someone hacks their firmware causing their car to always travel 5mph under the speed limit to aid fuel efficiency then they might not get caught, but we have no reason to want to catch them anyway.

Realistically, you handle such things via deterrence. When an accident happens, the car's software can be dumped (in some cases), and criminal/civil liability can be assessed based on the results. The risk of heavy fines or jail will probably keep most people from hacking their self-driving cars to be more aggressive.

Because that threat today keeps people from modifying their cars by cutting springs, installing lift kits, putting on ridiculous wheel/tire packages, and changing to poorly "engineered" HID retrofit lighting kits.

I see unsafe modified cars almost every week right now.

Realistically, the simple fact that it will be hard to modify in fashion that doesn't break the car completely is a huge deterrent. Of course it is possible for a single individual or small group to hack the software and then distribute it like consoles hacking

The economic case for hacking them is much different, the hacker risks prenablently bricking their device, in fact it might be nessacary to destroy a couple to learn about them. It is going to be harder to do this to a $20,000 device you rely on to go to work/school. Secondly the varsity of car models is going to dwarf the number of console versions. These two factor combine to make the hacking much more expensive and riskier.

It feels more prudent to wait and see if this becomes a problem and from that poin look at how/if to solve it. After all if a hack modifies a car's software to dive faster, accelerate faster, or something similar. You can do as the OP suggested and set up cammeras to record traffic and see who is driving a car that behaves out of spec and go arrest them.

Those kinds of modifications are of a completely different category than modifications to an autonomous driving system. There aren't serious civil or criminal penalties to deter "ridiculous wheel/tire packages" because they don't really have the same potential to cause serious problems.

I don't think it is really that complicated. On boot, a piece of embedded software (that can't be overwritten) does a checksum of the current software installed. If the checksum doesn't match the original, then it sets an internal bit to 1 to mark that the software has been changed. That internal bit can't be set back to 0, so the car is essentially marked as non-licensed and now can't be legally driven on the road.

Checking for non-licensed modifications is simply checking if that bit is set. Implementation would be a little more complex then the above, but similar technology already exists and is in use. The result is that modifications would be possible, but driving the resulting car on public roads would not.

A system for allowing new software to be placed on a car could be setup, but it would still require re-licensing the whole car so it wouldn't necessarily be an easy process.

This wouldn't solve that problem either. The way around it would be to make a look-alike module that replaces the current driving system entirely (hardware and all) and lets you swap them around during license checks. It'd also impede legitimate updates from the manufacturer, and violate laws that many states have that require that cars be fixable by non-dealer mechanics.

I'm talking about the likely legal framework that the systems would be implemented against. The technical details would of course be important.

But for instance, other systems could record and report vehicles with anomalous behaviors.

At some level, making an outward show of being a good person and being a good person are indistinguishable.