I might spend $100k on a house that I was going to live in for 20-30 years, even if I knew that at the end of that time it was going to be worthless. But I certainly wouldn't spend $500k+ on a house unless I knew I'd be able to sell it for more down the road.
But if you're putting in capital, you have to factor in the cost of money. There are far better things to do with $500k than to park it in an asset that doesn't appreciate.
And of course, you have to factor in the risk that the price will actually drop. Risk isn't free either: it doesn't make sense to take a risk that you may lose money, unless there's a counter probability that you will actually gain money instead.
Even if your monthly payment was less than your rent, you still might be better off doing something else. To get a good interest rate you need 75% LTV; which means on a $500k house you need to put up $125k up front. You might be better taking that $125k and investing it in an S&P500 index fund, and then paying rent, than taking that $125k and putting it into a house and making a mortgage payment. (Obviously you need to do the actual math here to see if it makes sense or not.)
<EDIT>And of course there's the diversification aspect too. If your net worth is $300k ($500k house + $50k other investments - $250k mortgage), it would normally be a really poor decision to have 166% of that ($300k / $500k) in a single asset, whose value could drop drastically for any number of reasons.
All that to say -- if your house isn't an investment, it had better be significantly cheaper than renting before it makes financial sense.</EDIT>
From a societal flexibility perspective, it's much better if most people rent; particularly poorer people who can't as easily from the economic shock of having their house lose all its value. If the bottom fell out of the economy in the city I'm in, and my house's price dropped by half, I could afford to cut my losses and start over. A lot of people are stuck.
I'm told that Germany is very different. It's not at all expected that people strive to own their own home; people rent the same place for decades and are perfectly happy. Because they've never had out-of-control housing prices (at least, not in the last few decades), they can keep things rational.
If the UK's policies suddenly looked like Germany's I'd probably take a massive haircut on the house I own. Personally, I'd be willing to do that if I knew it would make things better for other people; I'd be OK financially. But a lot of people wouldn't, so I understand why it's difficult to change.
We all need a place to live so a house serves a physical utility as well that other investments do not, in addition to being able to collect unlimited rent when the mortgage is eventually paid off eventually. I think there is a lot of upside to home ownership if it is financially feasible.
But was it ever an investment as an appreciating asset per se? I'm not aware of such a situation but I'd be interested to learn otherwise.
Sure, you can, but only if real estate in established neighborhoods is more expensive than housing in new neighborhoods, which isn't hyperlocally sustainable but might be sustainable over a broader region.
It's starting to happen in Austin: duplexes/fourplexes are getting built in some of the nicer, more central areas.