If you want to sort out this problem then you will need to address the concerns about "those people".

There are some very real concerns that people have. Violence. Drugs. Noise. School quality. People want a good environment in which to bring up their kids and these are all crucial. Fortunately they could be solved in other ways than expensive housing. Unfortunately people have a very justified fear that they wont be.