I work part time remotely, and we have other devs full time remote. I definitely feel like I'm "missing out" just being OOO part time, be it the hoc discussions or just going out for beers.

Of course, it only matters if you let it matter. Some people don't care or need that stuff. They worry about their own work and leave work at work, which I'm working on doing :)