Of course, I live alone in a decent-sized apartment outside of downtown. I also work at an XP shop (I actually much prefer remote pairing) and I'd probably feel differently if I had to work alone all day every day.
But in a way, this is not a balanced relationship. Sure, you'd want to interact with people like you who are excited about being together, so that's the preferred state anyway, but if most of your colleagues are of the 2nd sort, then you put the effort of going in the office to no reward. You could connect with people outside your immediate sphere, but team cohesion isn't helped by that and you might start disliking working there. So in a way you get some bubbles forming - those who prefer virtual interaction and those who prefer face-to-face.
This will probably be a big factor for career progression - is your manager leaning towards WFH or office-work? If you're on the other side, you might get left behind. Not that there aren't plenty of such possible discrepancies already out there
The people I want to talk to are usually busy doing actual work so we usually catch up at lunch or after work. I also have the personal numbers of those people. All the people that have time to talk during the work day I'd rather not talk to. I know this is basically an insult for that I'm sorry but I suspect you are one of those people. The awkward guy with no friends that comes to the office for his social interaction fix.
Riffing ideas is not helpful sorry but its just your awkward way of socializing. Has anyone really ever said oh ok I'll just trash this work and start over after "riffing"? No they just go "haha" and continue doing whatever they were doing.
So I'd happily pay the commute cost if I have to for my mental well being. If you don't need that that's fine. You can stay home.
I have a gut feeling (so no data) that for many projects a core group of physically together people will out perform a group of remote workers of the same size "All other things being equal" (which they never are).
I come from games, having designers/artists/programmers able to look over each others shoulders and "riff" off each other's ideas is invaluable. But I know there are other kinds of work that don't need as much interaction so YMMV