We're already okay with the possibility that a human might drop dead in the middle of an interstate and lose control and more humans might be injured or killed as a result. As long as it is not more likely that a self-driving car will drop dead, I don't see why it's any different.
(And this isn't even getting into things like the possibility of catastrophic failure being lower and the options for recovery being much higher. You can run two robots with redundant hardware; you can't run two humans quite as easily. A human who loses vision will lose their concentration; a robot that loses vision can use its last known positions of nearby cars to attempt to pull over immediately in an orderly fashion.)
It's the same principle. You try to do the safest thing you can, but yes, at some point a car will fail and be stuck in the middle lane of an interstate because the autopilot fails, just as can happen because of mechanical failure.
Which isn't really any different than any other sort of major mechanical failure.