Many homes in those areas are only occupied a portion of the year, by people with other homes elsewhere. Others are owned and rented out by family trusts.
I've talked to many people of mid-high means who are in or near retirement that have made such arrangements part of their financial and estate plans.
In London a lot of families own homes or have mortgages that are very small compared to current wages (so effectively owned if not quite yet).
They're only 'wealthy' because external observers are valuing their home at silly amounts. Generally they're of very modest means otherwise.
I think it's weird to call a family that bought a home 30 years ago in Inner London "ultra-wealthy".
It basically just shows how odd the financial system is that we're attempting to put values on family homes this way.
Incidentally this is the same reason why the Catholic Church is not rich. It looks like it if you value the land on which their churches sit at the development value they would fetch if sold... but running those churches is their primary purpose! So they could only realise that value by shutting down their only reason for existing.
The weird thing about living in San Diego is that it's incredibly expensive to live here, but there are THOUSANDS of people here who don't work whatsoever. It feels a bit like a retirement community, except for the ultra-wealthy.