The primary factor is concentration of capital: it is exponentially easier to raise money in the Bay. The second is concentration of talent and peers -- early customers, beta testers, development partners.
The Internet was supposed to make place less relevant, but as far as I can tell it's had a paradoxically opposite effect. When I was younger, I didn't have such a strong sense that you must be in one of maybe ten urban areas (but preferably SF) to do anything. Today it feels a bit like things happen in one of maybe ten cities, and everywhere else doesn't exist. Americans have always joked about the "flyover country," but I've distinctly noticed a massive increase in this effect since circa 2000.
SF, Seattle, NYC, DC, Boston, etc., are still the major magnets, but "hip" urban living isn't as confined to the coasts as it used to be IMO.
Some industries are very centralized in one city - tech in the bay area, finance in NYC, entertainment in LA, etc.
It's frankly pathetic that the so-called "biggest innovators" and people who "get the internet" feel the need to be constrained in an already crowded city.
Face to face communication is important, but there are tools to help with day to day communication
The Linux world knows much better how to deal with this then your average "Web 2.0" company