The opportunity cost of the capital is really important.

However, many jurisdiction give tax advantage to owner occupied property.

Eg you have to pay all kinds of taxes on the return from a million dollars in the stock market, compared to the housing services you consume from your own million dollar house.

(That's in most countries. In eg Singapore we don't have capital gains tax. So it's a bit more even.)