For one it's an extremely capital intensive operation for most folks. In fact buying a property is usually many times their net worth. Which means now one is not allowed to make mistakes, because losing means not only going to zero, but going underwater many times over what the cashflow can cover.

This argument falls apart once a property is a fraction of someone's net worth, but it's rarely the case.