They may be more like classic cars than regular vehicles at some point, though.
You could have picked a better example. I see tons of young people smoke and wonder what the hell is wrong with them, whether or not they haven't been paying attention for the last 40 years and then I realize they didn't because they weren't there to begin with. So the tobacco industry can work their nasty charms on them with abandon because there are new potential suckers born every day.
Or alternatively, maybe as strange as a motorcycle or vintage MG.
I think people underestimate the cultural impact that self-driving vehicles will have - imagine a whole generation or two after self-driving vehicles are generally available - how many people will bother learning to drive? I think it might become more of a job-specific skill than a general 'adult life' skill as it is now in most places.
At some point, human driven cars become novelties, just like that. There is no reason to ban them, but as you suggest, maybe there will be some HOV-like lanes where they don't really have access. Or even some time constraints (not during rush hour on some key roads, not in lower manhattan, etc).