>They are opportunities to build friendships and have some fun with people I wouldn't mind knowing better.
Which I would argue is very clearly a benefit to your ability to work together. My team off-sites certainly include formal information exchanges, but we specifically try not to spend too much time of things that could be equally well communicated via email or video conference.
I love the idea of an annual face to face event, but none of the benefits I want from it are work related. They are opportunities to build friendships and have some fun with people I wouldn't mind knowing better.