Buses are lacking privacy/familiar feel of your own van (considering you want to use one car for a larger amount of people). And it's harder to plan a route the more people in the vehicle.
Now you have a point about kids safety. The car could still pick up and move things while you're busy, this means less time on the road when you are aka rush hours.
Isn't this a more likely scenario if the car was leased and/or it was a car pooling thing? But in that case, why are we wasting our times with cars? Why aren't we making buses self-driving? Car pool even more people in one vehicle!
I can imagine a common type of vehicle for self-driving cars that will be popular would be 'small buses', ones that carry 8-12 people, that drive unique routes.