This is a really good point.
To remove the moral overtones, I like to think of selfishness as local vs global optimization. The prisoner's dilemma is a great (abstract) example where "locally optimal" or "selfish" behaviour creates a situation where everyone is individually worse off than they need to be.
In the real world, this kind of situation is especially visible in urban planning and traffic control. Things like lane widening or adding a new road can famously worsen traffic congestion. Lane mergers is another. The "zipper merge" is known to produce shorter queues and fewer accidents, but individuals have motive to "cheat" and merge earlier in these cases, which ends up causing worse traffic for everyone.
Personally, I find this game theoretic view point of "selfishness" to generate more precise and relevant questions than the more morally-tinted stances. Maybe that kind of thing is up your alley.
If, as a starting point, someone believes the population is growing exponentially then they likely also believe it doesn't matter if they try to be selfish or not. There will be a lot of people with limited resources and that is probably going to reduce their quality of life.
Now there are a lot of issues with this view and the actual reality might favour anything, but I don't think the average selfish or selfless person has thought about them. So I'm pretty happy to excuse both until I see arguments with more facts than 'I think' or implicit 'selfishness is bad it sounds unfriendly'.