Then you get issues with the rich from other places using your city as a wealth storage zone and suddenly you have millions of homes empty but no one has anywhere to live so the city dies.