I saw an interview with Bill Gates where he said much the same thing to a comment about his generosity. He pointed out he doesn't sacrifice a single item/experience he wants in life giving away what he does. And a ordinary person that gives a few hundred gives up more than him in terms of what it costs them.

Its a valid point as long as it doesn't take away from the fact the other is also generous.