That's a fair point, and this is merely my opinion, but if I could afford housing all over the world, I certainly wouldn't want to deal with the fact that I maybe cannot go visit London/NYC/wherever on a whim and stay at my own private residence. Renting has downsides even with "perfect renting arrangements" which cannot be mitigated by a third party (well, without presumably breaking an entire host of laws).

That's kind of what I was jokingly suggesting; that the caretakers - who likely are the only ones who see the inside of the place for years - just surreptitiously rent it out from under the actual owner.