It becomes even easier with a bigger pool.
Also, it breaks the driver/car connection since you are no longer a driver.
Finally, the real reason. Without a cab driver to pay and tip, cabs are actually cheaper than ownership. This is Uber's end game.
Car ownership requires at worst 1 car per person.
Carpooling requires a lot of planning and only works for with individuals with similar scheduled routes and requires the humans to include the time to drop off additional passengers or pass control of the vehicle to another for additional legs of the trip. Totally aside from the logistical nightmare it requires people to trust others with an asset worth thousands or 10s of thousands of dollars.
Imagine you pick 10 people. Getting them all to and from work shopping etc requires 10 cars at worst in the ownership strategy.
Imagine that carpooling might reduce it to 8 or 9. Its extremely likely that a route could be plotted for 3-5 cars to take all to their desired destinations both by moving back and forth for trips that don't coincide and by making slightly longer trips that move multiple parties.
Now imagine that you have the entire population and can match all the rides that coincide and include an entire range of vehicles including buses to get everyone from A to B maybe you can reduce this to 1-2 per commuter.
A fleet of taxis or cars from similar services could do the same. I don't see how the self-driving aspect changes the situation, other than possibly reducing cost by eliminating the driver.
Self driving cars by displacing the driver may eventually be cheap and flexible enough to compete with ownership.