You have the right perspective on this IMO. Thinking about owning a home as an investment is foolhardy: very few people are going to get rich off an increase in their home's value, and only a few are even going to significantly increase their retirement funds after downsizing.

Houses are places to live, not investments. It's in all of our interests if they're as affordable as possible everywhere instead of pushing the equivalence of an arms race.