They way that would work is that when you car is in semi-manual mode, it tells the others cars that a human is controlling steering and speed. The other cars will then give the human extra room (e.g., no drafting or close formation driving on the freeway, no crossing at right angles in an intersection at full speed just inches apart, or that kind of stuff that the cars will do when computers are driving all of the cars in an area).
The automated system will still be monitoring the human and can take over if the human does something bad, so the other drivers don't have to worry that a human driven car will suddenly cross the line into their lane and kill them.
This could even be turned into a game, where the automated system can score the human driver based on how well the human drove and how often the system had to intervene to keep the human from crashing.
Note that racing could also be allowed in such a system. Want to take that Aston Martin out for a race against your neighbor's Ferrari? Let the system know, and it can give you a course through the city, clear a bubble around you two, and let you go at it.
With a system like that you could easily design a market where vehicles can choose to pay a toll to get a favorable path through traffic. You might see a few options and incentives given for choosing a more fuel-efficient path, etc.
By the time all personal transportation is automatic, it won't just be the status quo with "robot chauffeurs". The whole transportation ecosystem will have to change substantially. Self-driving cars are just a first step to get people used to (or even enamored with) the idea of automatic transportation on a large scale.
I think horse riding might be the best example of the way things will go if and when automatically driven cars become a reality. Once a life skill required by everyone, now a hobby in all but the most extreme of communities.
[1] http://www.autoweek.com/article/20120803/carnews/120809942
6M police-reported crashes in 2007. 1.71 million police-reported crashes with an injury. 37,000 traffic fatalities in 2007, 30,862 in 2009.
Traffic fatality data is here with only some high-level info about non-fatality crashes: http://www-fars.nhtsa.dot.gov/Main/index.aspx
What sort of accidents will GoogleCar be prone to? Fender benders, or a head on at 65mph every 500K?
I'm very hopeful about this. Most people don't want to drive most of the time. As long as I can still drive my V12 Aston martin manually when I want to, this is awesome.