Autopilots immediately get rid of that so I would disagree and argue the less humans the better. Have fun playing around with the "politeness" lever for example:
Traffic lights are a bitch, changing lanes is inevitable, incidents are inevitable.
That adds up very quickly, and when you consider the presence of intersections and traffic lights, you aren't dealing with an isolated roadway where cars just...drive. There are very real limits on how many cars you can stuff in a given stretch of road without radically redesigning roads in a way that makes them entirely unusable for pedestrians and cyclists (i.e. replacing traffic lights with slot-based intersections[0]), meaning there's an upper limit to how much self-driving cars can limit congestion under the most optimistic scenarios.
When self-driving cars eventually manage to take hold, no one reasonably expects car traffic not to increase. Right now, you park your car and do things. There's no real opportunity cost, because you can't be two places at once. When your car can drive around on its own, there's now a real opportunity cost. If it can earn money for you with no human present, there's going to be a massive incentive to let it do so. Stores will use self-driving delivery vehicles--either their own or as part of their own delivery sharing services--for deliveries.
Will there hopefully be big improvements in pedestrian safety and decreased accidents? I think so, though they certainly won't eliminate either. But there will be an incredible amount of increased traffic that goes along with it.