By buying a house, you've put upward pressure on house prices in your area, thus making them more expensive for everyone else. Some people would be nudged over the edge from affording it to not affording it.

I think the loss, if it actually exists, is distributed thinly over a lot of people so it's hard to see.