In many places there are rules designed to make housing more expensive, such as requiring houses to be bigger than they need to be. Poor people living near you reduce the value of your house, so there's a strong incentive to make new housing less affordable.