> if enough affluent people leave, property prices will tank and you will be paying off a mortgage for a larger sum than what the property is worth.

Or, as California is a non-recourse state, leaving the bank holding the bag, paying only taxes on the unpaid debt.

> Meanwhile, all these affluent people have to move somewhere

Well, they actually don't have to move, and if they do it may not be to where you predict.