Well, you're not taking those people's money in the first place. The justification for including them in both calculations is so that there are no discontinuities in the system where producing a little more of value for others means taking home a lot less in resources for yourself (like many means-tested programs).