Actually, if you could build 1000 units of really high end housing in place of, say, 100 units of average, low-density housing, it would still help, because it lessens the pressure on some of the existing housing.

That's the point, to increase the supply.