For the privacy factor, I think people will just get used to it as they have with current public transport, or tiers will be in operation (as you suggested) or someone will inevitably design a pod-based vehicle with seats facing port and starboard and a clever door system.
I do wonder if people will be uncomfortable with other passengers knowing exactly where they live and work? Perhaps designating pickup from the nearest corner would help with that, and pricing that encourages that (to save extra turns down side streets) will help too.
I wrote a bit more about all of these likely developments here for anyone else fascinated by the rise of automated vehicles: http://www.isaacforman.com.au/n/self-driving-cars/
Your phone could also learn your departure time, and automatically determine when there are cars available / pricing is optimal to tell you to leave.
Extend this thought to its logical conclusion, and you realize that (self-driving) commuter busses and light rail can be programmed into the network as plain-old cars that happen to have high occupant capacities. You set that you're willing to take a bus (and one is en route), it'll stop for you. You set that you want a less noisy/drunk/homeless-filled experience, and you get a car instead.
You could also set a price threshold, where everyone in the car must already be willing to pay as much as you are to be picked up there and then. Low threshold—a taxi-like sedan comes. High threshold—a limo.