Investment buyers and owner-occupants both provide the cash to the homebuilders to allow them to go build the next home. You can't sell a home without a buyer.

I don't think requiring anyone who wants housing to be financially able to buy an entire home is sensible, meaning that some entity needs to be in the business of paying for more housing than they personally need to allow people to have housing who can't or don't want to buy an entire dwelling unit.

Maybe you want the government to have a monopoly on that activity, but someone needs to provide it or you end up with far less access to housing and far more people living on the street than currently.