If team building, then yes. But charity work doesn't really sound like team building. It sounds more like "look how good our company is we do charity work on company time" culture building.
In which case the goal is to make people feel good about the company.
2 birds, 1 stone
(https://en.wiktionary.org/wiki/kill_two_birds_with_one_stone)
Sending people home defeats the purpose.