Perhaps that's the disconnect. Your odds of being physically assaulted by an addict or mentally ill person in SF might be low, but your odds of being verbally accosted or having your personal space invaded are much higher, which contributes to the feeling of being unsafe. (And is itself a problem.)