You have multiple groups of drivers including those who appreciate self-driving assistance on long boring highway stretches and may not even drive in cities that much and those who never want to get behind a wheel or perhaps ever own a car.
You could force the car to avoid certain intersections with lots of close calls, or give the car advance info about unexpected objects detected in the road by cars up ahead, etc. which would make the task a little easier.
Apart from regulations, level 4 doesn't require a remote operator monitoring at all times. It just requires them to be available when the car recognizes it can't handle the situation. Depending on how frequently interventions are needed, that could mean a remote operator for every car, or one for every 10 thousand.
The important part about Level 4 is that the car recognizes its own limits and doesn't act outside of them. How practical those limits are is completely undefined by SAE. Level 4 is a continuum whose limit is level 5. We will definitely be seeing practical commercial value as we approach level 5 but haven't yet gotten there.
It's basically level 5 or bust.