We're already heading down that route [1]. Although I can't see it ever being an offense to manually drive your car, the car will just automatically take over the driving duties if it doesn't like the way you are driving.

[1] http://www.autoweek.com/article/20120803/carnews/120809942