I really wonder how much of it is this, and how much of it is virtue-signaling at the corporate level: "look at how great our company is! We have our employees spend work hours on charity work!"
While it wouldn't do much for "team building", it seems like society would be better off if these companies simply contributed to a fund to pay minimum-wage workers to do this charity work. This would get more charity work done, and also provide jobs for some underprivileged people.
As for your soapbox, I completely agree.
It used to be that religion provided these answers. Believe in God, have faith, go through these rituals, help the poor and you're a good person who has successfully discharged moral responsibility.
Religion has faded away, so we're left with "a good person helps the poor" which everyone agrees is a pretty moral thing to do. But the more indirect it gets the less moral value is assigned to it.
People assign moral value to hours of time invested because time is what we all have equally. Time is not money when it comes to judging character. It must be this way because otherwise it'd be that very high earning people help the poor to a vastly greater extent than the average person does, just by turning up to work every day and paying their taxes. If that automatically made them morally superior to everyone else, it'd create a lot of resentment. No matter how hard you worked you would never be as "good" as the rich guy who just turns up and does some lucky stock trading or whatever.
I think sometimes programmers don't feel this way to quite the same extent as other sorts of office workers because we sometimes get more meaning from our work than someone in e.g. sales. The Silicon Valley stereotype of "Join our startup because we're changing the world for good!" has a basis in real self-perceptions. If you believe you're making the world a better place by actually doing your work, of course doing extra charity stuff will seem silly and a poor allocation of time/money relative to other ways of achieving the same ends.
This would only make sense if paying taxes actually helped the poor. In America, it doesn't really follow. There's not much of a social safety net here. For proof, all you have to do is walk around various large cities' downtowns and see all the homeless people. There's thousands of them within a very short walking distance of the White House and Capitol buildings in DC.
Sure, America is very large. It may not be 'much' of a social safety net relative to other countries. But it's still the case that the rich subsidise the poor to a very large degree, in any western country.
There are some other components administered by the same agency, including TANF, SCHIP, and Medicare, so yes there is some safety net there, but relative to other western countries it's rather poor. If it weren't, we wouldn't have so many homeless people. America is NOT like western European nations.
In the end, all social security is a wealth transfer from rich to poor. Even rich employees become much (money) poorer when they retire. That's not a criticism, just an observation.
Of course that doesn't do much good for the company's employees that don't wind up volunteering / doing the work, perhaps even increasing the separation the employees may feel toward the less-privileged. Service projects, mission trips, etc. can do a lot toward increasing humility, gratitude, and a spirit of giving in a more-privileged person. One could argue this is partly why companies want to do this sort of thing, so their employees grow (among other good / less-than-good reasons).
Perhaps a company could split the difference. Hire others to do the manual labor that day, but take half the company's team to the jobsite do the work with them. Then on another day you do the same with the other half. That way, all parties involved receive some benefit, and everybody is getting to meet a human being behind the statistics. Just a thought.
P.S. It's a certain level of irony when a party steps in to "help," when in reality they're taking away opportunity from parties more prepared for that sort of work, and are probably doing a worse job, aka building shoddily-constructed houses for the poor. This is especially a problem in developing countries, when an organization or church flies a bunch of inexperienced people into the country, brings in supplies, etc., and virtually takes over the available construction projects in an area because they're doing it for free. All of a sudden the local tradespeople have their businesses dry up, and they may not be able to recover -- and they're the ones that the area needs to continue the work afterwards! A far better method, in my opinion, is to get the locals involved and ask how you can help. Volunteerism is fantastic, but we should do it wisely. </soapbox>