I grew up thinking that every time you walk through a dense urban neighborhood, you have to be on your guard lest you get knifed or shot or worse. Ironically, I didn't get this from my parents; my mom used to ride the subway alone into Manhattan for junior high, and my dad grew up in Zamboanga City. But when your hometown has only 13,000 people and literally zero crime, you're shuttled by car everywhere, your peers all live in the suburbs too, your only image of the city are the news reports about murders & drug dealing, and your one urban friend is a black dude from Dorchester who regales everyone with stories about how somebody got shot on his front step and that's why his parents insist he wake up at 5:30 AM to get bussed to a suburban school, you get a pretty biased perspective.
There's a pretty strong selection bias maintaining this illusion too: people who are still in the suburbs for this reason will never come in contact with urban dwellers who aren't actually murderers, out of fear, and similarly the majority of urban dwellers aren't actually going to move out to the suburbs (and even if they did, their past lives in the city are just abstract tales to people who haven't experienced it). So much like other geographically-based stereotypes (like "immigrants are all bad hombres", "California is falling apart", "suburbanites are all boring Stepford Wives with nothing to do but gossip", "Appalachia is filled with uneducated hicks", and "religious people are wackos"), this remains a stubbornly persistent belief despite the reality.
The parking garages that indicate the number of available spaces on a level are a good counter to this: I find that most able-bodied folks will see the bottom few floors have few spaces open and go for a higher one.
Younger people who aren't in (have graduated) college seem to be a little more reasonable.
As a native Lynchburger who subsequently spent about 15 years in the Raleigh area before moving to San Jose a few years ago, ... I never felt this way growing up in Lynchburg. Yes, Lynchburg is approximately 50% white / 50% black, with approximately 0% any other ethnicity, and the majority of the white population is either exurbs or specific segregated parts of the city. But even though I grew up in Boonsboro I went to RS Payne Elementary downtown and then EC Glass for high school ... in HS, I freely wandered around that part of town after school (usually while waiting for sports practice of some sort). I never felt endangered. Nor, afaik, did my [mostly white] friends.
That's a stupid anecdote, but I think it actually used to be better than it is now, and that mass & social media have created so much fearmongering that more people are hyper-insecure and wary of any unplanned interaction with anyone not in their in-group.
It's probably exacerbated because for people who grow up in low-density areas, the idea that the vast majority of people will just ignore you is a foreign concept. If you're in a small town and you see someone you know & like, you go up to them and chat. If you see someone you don't like, you bully them or get bullied. If you just ignore them, you likely get moved from the "like" to "don't like" category. If you extend this model out from a town of < 10K to a city of 1M, it collapses very quickly, but not before generating some very erroneous predictions.