If we remove the regulatory barriers to building more housing, the rich and the less rich can buy as much housing as they want.

Think about this in the context of any other good. No one worries that if we build more cars, they'll all be purchased by rich car collectors. Yes, when more cars are built, a few very rich people with many cars will buy more. But the increased supply also make it more economical for everyone to buy more cars.