I'm also skeptical that we will reach the point where cars will not need a driver at all in the near future. I might be wrong but it seems to me you need something close to AGI to have a fully autonomous car.

And with current approaches to self driving cars you need lots of data, sensors, fast and reliable internet connection, accurate maps and so forth, all of which are not at the level needed to have autonomy on all roads. So even if the algorithms would be good enough you'll need much more than that to get to self driving cars.

On the other hand a human only needs one working eye and a way to steer the car to be able do drive pretty much anywhere.