Voyage.auto [2] claims to be deploying such cars now. Or rather, their web site contains announcements from late 2018 that they were doing so. Later information seems to be lacking. It's a basically good idea, but they claim an on-site staff of 5 for three cars, so they are nowhere near this making financial sense.
A real problem with self-driving cars is the false-alarm rate for emergency braking. If you're conservative about crash prevention, every once in a while the vehicle is going to detect something it doesn't identify as safe and will brake hard. That's why Uber turned off automatic braking in their cars - "to reduce potential for erratic behavior."[3] False-alarm braking, or even strong braking conservative by human standards, limits customer acceptance.
[1] https://arstechnica.com/cars/2015/05/googles-quirky-self-dri...
[2] https://www.latimes.com/business/autos/la-fi-uber-arizona-nt...
I work for Google, opinions are my own.
I don't know the specifics myself but I trust what you're saying is true and agree those are real problems.
Nevertheless, one of the best things to do when you have a really hard problem is to simplify the problem. It's not as though we have to have fully self driving cars before they're released.
I think what would make a lot more sense is some middle ground where we have certain sections of the road where self driving cars will be able to work well and only allow them there.