I am picturing this more as an extreme edge case than anything else. If a person can safely do 20-25mph in an area then a self-driving car should be able to do the same thing.

It's the edge case where a car needs to cross a really high traffic area that's going to be an issue and people can't do 20mph right in front of a high traffic box store entrance etc. Further, going slow for a few hundred feet really does not take that long.