At this point, "it's easy for a person to do it" is probably better evidence that it's a hard problem in AI.
In this case, I believe it's an inverse learning problem. We observe other agents acting in an environment, deduce their policy, then implement that policy ourselves.
A self-driving car that can reconfigure it's software to mimic the behavior of other cars it observes, sounds difficult to me.
This is essential, especially internationally. If you're an American you think it's perfect normal to be allowed to turn right even when the traffic light is red. You don't get told not to do that when you hire a car in Italy. Now an automatic car can be programmed with different rules depending on which country it's in, but even in the same country, you wouldn't enter a box junction when the exit isn't clear, at least in most places. Turn up in rush hour London and you find there's no way to actually get through the junction unless you do that, because others fill the junction so when your light is on green, the exit isn't clear.
I remember pulling upto a red light late an night. Empty road, but the red light only goes green when it detects a car. It didn't. What would a fully automatic car do in that case?
Never is a long time, but I don't see self driving cars coping well when traffic is coming the wrong way down the street, let alone when cars are driving along the sidewalk
A car won't get through a block as quickly as it could if it merely obeys the laws makes sure it doesn't run into anybody. I'm sure this will cause great joy among people crossing the street and consternation among other drivers but the problem isn't a blocking one as long as the car can actually avoid hitting people - even if it's more cautious than it needs to be.