However I find that the amount of "heads down" work I get done is greatly diminished when I'm in an office. And, much worse, there's a lot of noise in that added communication of being in the office. Remote teams, in my experience, have dramatically less "office politics".
Office space is great for communicating "big ideas" but these aren't anywhere near the bulk of communications being had. For most of the communications needs of software remote works fine (in my experience better).
I work on a quite a few "big idea" projects and I've found the best solution is to visit the office quarterly, get all the big idea brainstorming done, then scurry off to my remote office where I'm not distracted by office politics and can just get things done. A little face time goes a long way, and annual, or semi-annual all hands meetups can do wonders at filling in the gaps created on remote teams.
In particular, I think distributed teams work best when a) people already have strong relationships built via plenty of in-person interaction, b) they meet regularly in person, and c) the nature of the work is relatively predictable.
There's definitely some work like that, but early-stage startups benefit a great deal from in person communication, which is high bandwidth, low latency, and very high quality.
I'd prefer private offices to remote distributed work. But I find it's actually easier to find remote jobs than software companies that make an effort to provide private offices.
If something like this seems obvious to you, and obviously a lot of smart people are acting in the opposite way, maybe you should re-consider your assumptions.
In reality, online communication tools currently are a poor proxy for in-person communication in many situations. Online tools do not pick up the array of non-verbal cues humans make when talking to one another, are not stereoscopic, do not allow freedom of motion, do not permit the collaborative use of 3d space, etc. They are a piss poor real time communication medium compared to an in-person conversation in a room with whiteboards. If you can never actually have a face to face conversation with someone on your team, there's an entire dimension to communication you are completely prevented from ever experiencing with that person.