40K people a year die in the US from vehicle accidents. Why would you not ban humans from driving if machines were objectively superior? Where do you find the "right" to drive to be? Driving is a privilege granted by the government, not a right.
If fully-automated vehicles become commercially available (which seems likely at some point, though not in the immediate future), we will likely see rapid adoption in the applications where they are very well suited, followed by a long, slow, gradual trend toward broader usage. The hardest cases will be the ones sticking around longest, and one can't simply legislate them out of existence. Just replacing a small percentage of the fleet won't come anywhere close! It's only when automated cars have become so dominant that only a small percentage of the fleet still depends on manual drivers that it will become practical to start talking about a ban on manual operation.