>They are opportunities to build friendships and have some fun with people I wouldn't mind knowing better.

Which I would argue is very clearly a benefit to your ability to work together. My team off-sites certainly include formal information exchanges, but we specifically try not to spend too much time of things that could be equally well communicated via email or video conference.