That is, it seems to me like the places fully self-driving cars will initially be most successful is places where access is limited or "messiness" is minimized - places like highways, where you could even have "self-driving car only" lanes - a huge boons for long commutes. In cities, though, there are a lot of Really Hard Problems, that from what I've read engineers are not even close to being able to solve, things like:
1. Following detour signs 2. Following a street cop when they are contradicting stoplights or other road rules. 3. Dealing with construction signs in traffic, like when a construction worker has those signs they flip back and forth between "stop" and "go" (or heck, they just drop the sign and wave you by). 4. Knowing the difference between stopped traffic where you should wait (i.e. traffic is just really slow) or move around (i.e. someone is unloading).
That is, it seems like the state-of-the-art will allow self-driving ASSISTED vehicles in the pretty near future, but completely self-driving vehicles, that can handle all the complex, messy urban situations that people can, is not on the near horizon.
Things like standardized signs for cars to process and construction workers to hold would already solve 3/4 of the problems you've already mentioned.
In the not-so-far future, that resistance will be broken by the introduction of the robotaxi. Ueber will be spearheading that one. Too capital intensive for the current "big" taxi companies in the medium term. Too automated for the regulators to defend the taxi drivers working conditions.