I lived in a small, ostensibly safe town in Sweden, but people would come in from outside to violate the trust the people had. You need to be good friends with your neighbor so they can confront or report any suspicious outsiders.