It'll be a while before the human is completely out of the loop, but Waymo has filed a patent for 'remote assistance in low confidence situations', and many of the companies pursuing robotaxis have signalled similar initiatives.
The handoff speed is an issue as is the fact that the vehicle would have to recognize a problem in order to ask for help. But, yes, some sort of remote assistance system would seem to be one solution to at least certain edge cases.
I suppose you can construct really elaborate corner cases that you might give a car a pass on. And maybe there are ways to make things like construction zones easier for cars to navigate. But, yeah, for the most part, the computer is going to have to handle any driving situation a typical human would be able to.