Even luxury units add supply to the middle and affordable parts of the housing stock. Today's luxury units are often bought and moved into by someone moving up from a mid-market place, the buyer of which is likely moving up from a 25 year old now affordable place.

(Alternately, if no luxury unit is built, many of those buyers will be forced to buy the best-of-the-rest if they want to move in, putting more pressure on the middle, which in turn puts pressure on the affordable.)