This is not true. The art market is financialized and tends to go up over time, but this does not result from art scarcity. You can make a market go up by making huge institutional cash buyers 30% of the buyer's market, and by getting even independent sellers to collude on price. Independent buyers in that market are irrelevancies, because they bought on credit borrowed against a house at an artificial value.
Also, that little property bubble we had in the first decade of the 2000s was not a result of a sudden population increase.
edit: also, it's important to mention that "if housing tends to go up over time" is within a window. It can lose all of that false value in a day, and the government will respond by directly subsidizing all of the institutional losers, and their stockholders.
Blaming housing investors for housing affordability is like blaming gold market speculators for gold being expensive. If the price of gold comes down, the speculators lose money. Same with housing.