I have not found that to be true, at all. Quite the contrary, I've seen that when there is no forced team building, people only get to know each other to a minimal level. They do not build trust, they do not learn each others strengths and thereby put a heavier weigh on people's weaknesses. A small mistake by a coworker can become "That guy sucks, can't wait 'til he gets fired or leaves", and small cliques and camps end up developing across teams. It can become quite dysfunctional, quite fast.

Of course, in a way this means I agree with you - if you are not someone who wants to be on a team that spends time and energy on the warm and fuzzy team building tasks... quitting probably is the right move. There will be other teams that work better for you.