The biggest problem was always when there were some people local to one another who would get together physically to tackle a problem. Either they wouldn't think to dial-in the remote team members at all, or they would but the discussion would be conducted in such a manner that it was impossible to fully participate (i.e., poor speaker phone, whiteboard drawings, projector w/o webex).
I think remote teams can be very effective, but it is crucial that the entire org be oriented around communication channels that give the remote team members equal footing. I almost think the ideal is all or nothing -- everyone is remote or no one is.
If I had my wish, we'd all be in one office or at least local to each other so we could meetup and chat more often.
My main problem is: online communication makes discussion much more complicated than face-to-face, so you don't contribute as much ideas when people brainstorm or otherwise solve creative problems. If you have a lot of ideas, that can be depressing.
I visit the office several times a year, mostly to share thoughts and socialize.