This would only make sense if paying taxes actually helped the poor. In America, it doesn't really follow. There's not much of a social safety net here. For proof, all you have to do is walk around various large cities' downtowns and see all the homeless people. There's thousands of them within a very short walking distance of the White House and Capitol buildings in DC.
Sure, America is very large. It may not be 'much' of a social safety net relative to other countries. But it's still the case that the rich subsidise the poor to a very large degree, in any western country.
There are some other components administered by the same agency, including TANF, SCHIP, and Medicare, so yes there is some safety net there, but relative to other western countries it's rather poor. If it weren't, we wouldn't have so many homeless people. America is NOT like western European nations.
In the end, all social security is a wealth transfer from rich to poor. Even rich employees become much (money) poorer when they retire. That's not a criticism, just an observation.
It used to be that religion provided these answers. Believe in God, have faith, go through these rituals, help the poor and you're a good person who has successfully discharged moral responsibility.
Religion has faded away, so we're left with "a good person helps the poor" which everyone agrees is a pretty moral thing to do. But the more indirect it gets the less moral value is assigned to it.
People assign moral value to hours of time invested because time is what we all have equally. Time is not money when it comes to judging character. It must be this way because otherwise it'd be that very high earning people help the poor to a vastly greater extent than the average person does, just by turning up to work every day and paying their taxes. If that automatically made them morally superior to everyone else, it'd create a lot of resentment. No matter how hard you worked you would never be as "good" as the rich guy who just turns up and does some lucky stock trading or whatever.
I think sometimes programmers don't feel this way to quite the same extent as other sorts of office workers because we sometimes get more meaning from our work than someone in e.g. sales. The Silicon Valley stereotype of "Join our startup because we're changing the world for good!" has a basis in real self-perceptions. If you believe you're making the world a better place by actually doing your work, of course doing extra charity stuff will seem silly and a poor allocation of time/money relative to other ways of achieving the same ends.