I think this is a classic philosophical problem. Unless you are a philosopher, though, I don't think it really matters all that much. The bottom line is that you are making a sacrifice for someone else, and that you aren't involving other people in it.

Doing good for others should feel good! And if that internal reaction helps us want to continue doing good for others, then that's also a good thing.

I think also we should be less concerned about labeling ourselves this way. I would not try to make "altruist" a part of my identity (or "egoist", for that matter). You are just someone who does good things for others, and -- incidentally -- that makes you feel good. That should be enough.