I haven't had much trouble working with one other person, like the first poster modern communication tech does it fine for me. People complain that physically meeting only scales to as many people as can fit comfortably in a small conference room, but I'd say meeting electronically doesn't really scale past 2 or 3.
Even with just a 4-5 member team, all hands meetings are indeed phone calls that drag on (we're getting a bit better at keeping them brief).
And "software design debates rapidly turn into a game of telephone"--oh yes. I may be wrong, but I fancy I have an excellent memory for the points and outcomes of such discussions, and it's endlessly frustrating to me to recap over and over when the same thing comes up next week, and the week after, etc.. Actually I don't know if that would be improved by being closer--besides this, the largest software team I've been on was 3 in person, years ago.
On the other hand, the other four or so members of the team all are on the west coast and have met--I'm the only one on the east coast. That's a problem for scheduling meetings sometimes, but I don't feel out of the loop or insecure. Part of that, no doubt, is that I'm not a founder--just a contractor. I care about the product as something I've put substantial work into, but it's not my baby and I'm not worried about someone stealing my baby away. There are four of us contractors building the product for the two not-very-technical founders. I guess that's another hot topic.
Regarding the office space, I agree here too. My 1st business we ended up moving office spaces and had an absolute BLAST decking out the new space. It felt like our geeky clubhouse and we'd often hang out in the evenings playing games or just chatting.
All that being said-- different things work for different folks. 37s is a fine example of a distributed company that seems to work just fine.
Context: I started my company in 2005 from Ann Arbor (which quickly became Chicago) with 2 partners in NYC. Here's what we learned:
* It is very hard to get funded if you're not all in one place. Also, my previous startup, from '99-'01, relocated everyone as soon as the A round closed.
* All-hands company meetings are very hard to do, because they take the form of interminable phone calls instead of lively back-and-forths.
* Software design debates rapidly turn into a game of telephone.
* It's much harder to guage the mood of the team; we lost 2 people because they were unhappy about stuff that might have been resolved had we caught it earlier.
* Solo founders in far-flung branch offices quickly feel out of the loop, which sets up a bunch of vicious cycles involving security, assertiveness, style, and communication.
We've made it work by growing through it, painfully; we now have so many people in just two offices that there's a cameraderie and common purpose (I'm dignifying it, yes). It's still more painful than it would be if we were all in one place. I haven't said this anywhere but here, but we'd have shipped 9 months ago (as opposed to weeks ago) if we were all together.
The company I worked for between startups --- Arbor Networks (a startup, but I didn't start it, and I was employee #40something) --- has a business office in Boston and engineering in A2. This, too, was a debacle. Engineering and management are "us vs. them" in the best of circumstances.
More points:
* No rent for office space is not necessarily a win. We could all work from home in Chicago. But having an office is great, partly for the same reason some people like wearing suits in NYC --- act the part, be the part, etc.
* People are often wrong about where they're most productive. If people are unproductive in an office culture, you may just have the wrong office culture. In Peopleware, Tom DeMarco lays out a great strategy for a skunkworks-style office to house a product team. Peopleware is excellent, read it!
* You're also missing out on the social aspect. How often can you all go out for dinners? When you have a shitty day and you're all pissed at each other, can you go out for a beer? (A teammate I yelled at once brought in a six pack at 2 in the afternoon, killing the work day and resolving the problem). Yesterday was my block party; everyone in my office showed up. How much cooler would it have been if the NYC crew was there too?
* Some really excellent players simply get lonely working from their home or in a tiny branch office. Some of them know it already and won't consider you. Some of them find out after they join and quit in the middle of a cycle. Obviously lots don't, but why cut your chances at all?