I think some people don't like being visible in a public space that includes "others". Maybe a feeling of being judged and feeling self-conscious as a result?

Go back 10,000 years and almost all people encountered would be people familiar to you. Any other instances would've been a potential stand-off. I'd guess it's a carry over from that.