Right, but the problem there is the insolvency, not being underwater (conceptually, you have the same problem if you are insolvent but owe a nonzero amount that is less than or equal to the worth of the property, though if its less you may be able to stay off insolvency by increasing indebtedness -- though that will probably take convincing a lender that your income/expense balance will change soon enough that you won't end up insolvent and overleveraged.)
As long as you can continue to pay the mortgage each one, nobody is going to come along and kick you out of the house.
The "problem" is that you can't sell your house for a profit to exchange it for an even more expensive house. Hence, the rest of us are on the hook to ensure that your house can continue to appreciate in price.