Fair concerns, here's my take: there will be edge cases for a long time to come, so there needs to be a way to handle this. The car needs to detect these out-of-distribution events and hand over control to a person. Now, if we require a driver with a license to be behind the wheel ready to take over in 100ms, this will obviously not be worth the trouble.

The first part of the solution I think is first to make the car fail gracefully in unknown situations, e.g. slow down curbside. This would be dangerous on the free way, but free way driving is also the simplest part of driving.

The second part is allowing human control of the car without "driving it". For example, the if there is a police directing traffic, the car can alert the occupant and ask what it should do. You could even "paint" out the desired path to take on the screen. Basically you're just telling the car where to go and not controlling the steering wheel, gas and brake directly.

I'm sure you can conjure up cases where this will also not be enough, but at some point when the tech is good enough and there are enough cars on the road, then people will naturally adapt to self-driving cars in mind. If you're doing road-side work and you know there may be self-driving cars passing by, you'd do your best to make the signage and cones as clear as possible. Perhaps laws will be passed to make it illegal to try to fool a self-driving car, or to put up new signs without first giving car companies time to retrain their models. It's a small cost compared to making all roads digitally adapted to cars using sensors and what not. It's also a small cost compared to the 1.3 million people that die in accidents every year.