You must take risks with everything. Owning a house, paid off or not, is risky in a multitude of ways.

At least monetary wealth can be diversely invested; so you have some control over it. Most of the risks that come from owning a house are completely out of your hands.