https://en.wikipedia.org/wiki/Psychological_egoism#Criticism...
You either need to admit that some acts are selfless, or you end up just defining "what someone selfishly wants" as "what someone chooses to do", which isn't very useful.
There are a couple ways to handle this.
First, humans are not inherently rational. How many times does a human desire one goal but performs something that moves that goal further away from them, perhaps to the extent they destroy any hope of obtaining that goal?
Second, there are social benefits to selflessness that we will tend to be able to enjoy the benefits of but which are not always going to be there.
Consider a bird trained to press a button for food though the food only has a chance of being produced. Even in cases where there isn't food, the bird is still pressing the button to get food. Even when an outside observer knows the next press will not generate food, the bird is still pressing it for food. Even if we could inform the bird that food would not result from the next press, the bird would likely make the decision anyways to disregard our advice and press the button for the sake of food.
How crazy would it be to say "I'm going to build an AI that is selfless and altruistic. When it performs an act of altruism, it will be penalized."? That's not going to end well.
Do we want a world in which it doesn't make you feel anything at all, but by sheer force of rationality our apparently-disembodied consciousnesses choose to do good things? Well... presumably they're going to do it because it creates good outcomes, not merely because altruism is somehow intrinsically good in some abstract, unquestionable Platonic sense.
Altruism just doesn't work as its own moral principle somehow abstractly severed from its effects. We desire it precisely because it has good effects. This is not separable from the desirability of altruism... if altruistic behaviors simply had bad effects we would not continue to advocating for them anyhow because altruism was simply its own moral good, we would call those actions evil.
I'd propose that "true selflessness" of the sort that many people seem to propose in this sort of discussion is merely a fuzzy concept that has no possible manifestation in the real world. Clarifying the concept of "true selflessness" clarifies it it right out of existence.