If you look at any crime heat map of a city, you will note, hopefully with no surprise, that crime rate is inversely proportional to income distribution. If low income housing enters an area, the crime will go up. You may be fine with that, but you have to imagine that the already established residents of that area might not wish their crime rate to increase. Calling those concerns selfish is just name calling.