The first part of the solution I think is first to make the car fail gracefully in unknown situations, e.g. slow down curbside. This would be dangerous on the free way, but free way driving is also the simplest part of driving.
The second part is allowing human control of the car without "driving it". For example, the if there is a police directing traffic, the car can alert the occupant and ask what it should do. You could even "paint" out the desired path to take on the screen. Basically you're just telling the car where to go and not controlling the steering wheel, gas and brake directly.
I'm sure you can conjure up cases where this will also not be enough, but at some point when the tech is good enough and there are enough cars on the road, then people will naturally adapt to self-driving cars in mind. If you're doing road-side work and you know there may be self-driving cars passing by, you'd do your best to make the signage and cones as clear as possible. Perhaps laws will be passed to make it illegal to try to fool a self-driving car, or to put up new signs without first giving car companies time to retrain their models. It's a small cost compared to making all roads digitally adapted to cars using sensors and what not. It's also a small cost compared to the 1.3 million people that die in accidents every year.
Sometimes it seems like software developers are just blind to what happens in the real world and expect others to conform to their idealistic notions. Not going to happen.
It's simple: 1. habits change because people can foresee the consequences of their actions 2. regulation and laws
The benefit of cars outweighs the rights of pedestrians, just like the benefits of self-driving cars outweighs the rights of work crews to do whatever they feel like.
First of all, it's not a "pure ML approach" as in end-to-end. It's separated into perception and planning. As you can see in the UI now, the perception part is already quite good and gradually getting better. Adding lidar at this point wouldn't help much. What I do think will happen is that they will continue with the philosophy that the car should be able to drive without HD maps in most places (as if driving the route for the first time), but that they will augment the planner with low-res maps generated from other drivers in the hardest spots (e.g. weird intersections that humans also struggle with). They already have the capability to create these maps as seen during AI day, it's just a matter of admitting they are needed in highly complex areas.
The second misconception is that the planner is somehow making instinctual decisions based on a simple neural network. Their approach is similar to AlphaZero, which includes Monte Carlo simulations with deep neural networks to predict paths of everyone on the road. I see no reason why AlphaZero cannot eventually beat humans at predicting traffic as well as it can predict moves in Go and chess.
A third misconception I see is that there are so many disengagements, so it's clearly not safe. And that's true, it's not safe yet, but Tesla's strength is that they can have thousands of enthusiastic beta testers willing to drive around while paying careful attention, often intentionally seeking out difficult edge-cases. As the number of disengagements and the severity of them goes down over time they can roll out to even more users. Eventually they'll be safer than humans on average. I think that is likely to happen in a few years based on their rate of progress. What happens at that point is completely up to politics and human psychology.
In summary, if you actually go through and watch a couple of videos, I challenge you to find examples of disengagements that are complete show stoppers that can't be solved within 2-3 years. Also, go check out videos of earlier version so you can get a sense of the rate of improvement, which really is the only thing that matters (and it's massive). You can argue that they will plateau somewhere below human level, and that's certainly a possibility, but I wouldn't bet on it.