I don't think "perpetual growth," is really what people complain about. what they really mean is that they want to shortchange everyone who does not already have housing.

There are a finite number of people with finite needs. Growing the housing stock to house this finite number of people is not "perpetual growth" and it's downright offensive to say that when we camps of homeless people, camps of people in RVs without other housing, camps of people in their cars, people working full time but only able to afford overcrowded housing, and at the same time enormous wealth within miles.

A society that allows such destitution while others have enormous wealth is not one that is running well. It shortchanges us all, economically.