Even the first few help. You have rich people taking up houses and apartments made for middle class (or possibly even working class), as only rich people can now afford to live in them. If there’s more housing stock, even “luxury” units, they can vacate the lower end units, making room for middle class.