...if there's ten of them. If there's 1000 of them, now the rush hour traffic is both ways. That is the expected scenario, not "I have my very special car like nobody else has."

The self-driving cars are not in a hurry to get to their parking space. The commuters don't want to be late for a meeting, but the self-driving cars don't care about being late for a parking space.

Yes, and...? Doesn't change the fact that they're creating congestion (not only for themselves but also for other road users); plus the return trip can be expected to be "pick me up at $location at $time", which is again a scheduling problem.

Clearly, the solution is to let self-driving cars drive into town on the opposite side of the road.

Rush hour traffic in that single direction might get heavier if the drivers of single occupant cars can do other things with their time. It simultaneously removes some of the disincentive of a long commute, and also removes much of the benefit of not being the driver.

One common mitigation for rush hour traffic is to steal some lanes from the reverse-flow traffic to give more capacity for peak-flow traffic.