This is a very similar argument to "piracy is a moral ill because it deprives corporations and artists of money that would otherwise be spent on them." And it falls apart for similar reasons.

When more people are capable of buying houses, more houses are built to meet the demand.