I experienced this in the small town suburb I lived in once. There were always random kids in and out of the house, we'd give them food/water/bandaids whatever. Tell jokes/stories. Even though we aren't related, we all knew each other and our kids just kinda bounced around together. Definitely a cool experience.

I think the media making the entire world out to be some bogeyman waiting to kidnap your child is as much to blame as anything.