Nightlife, public transportation, grocery stores and restaurants are not non-existent in the Peninsula and South Bay. And if you live in one of the downtown areas, most of which are clean, safe and vibrant, all of these things are accessible by foot. Rents in these downtown areas are generally no more expensive than rents for comparable housing in desirable neighborhoods in San Francisco, and in many cases, the rents are meaningfully lower.
This said, it's worth considering that a lot of people prefer not to live in a dense, urban environment. One of the most attractive things about the greater Bay Area is that it offers different kinds of environments for different lifestyles within, by US standards, a relatively small geographic area, so there's something for just about everyone. What's wrong with that?
How would you "change" it? It's just not dense enough to support enough grocery stores/restaurants that everyone could walk everywhere. There are small downtown areas of each city where that may be possible but those areas are going to be expensive.
Given the absurd cost of housing in the bay area (http://www.slate.com/blogs/moneybox/2013/04/11/san_francisco...), you'd generally just reduce or eliminate height restrictions, minimum building setbacks, and parking requirements. At that point developers will build structures that increase density and support grocery stores / restaurants.
Out of the top 10 largest cities in the US only three of them are livable without a car - NY, Philadelphia and Chicago. (San Francisco is not in the top 10 for size)
Real estate developers like to build cheap and fast. Urban planners do what the book tells them. Public transportation is politically unpopular. And American's love, just love, their cars – don't do anything to get in the way (bike lanes, dedicated bus lanes, congestion pricing).
I do think that younger American's are more urban and are looking for a change. So we shall see what the future offers.
Just my two cents. Demo: 30, male, owns a car but hates it.
EDIT: Our bike lanes are coming along very well in downtown Chicago, and almost everyone loves the new Divvy bike sharing program.
Virtually nobody in our (sizable) office drives to work besides me, and I really have no excuse, since I'm a short walk from the Green Line. Public transportation is spotty in the suburbs, but in terms of coverage (both in space and time) the actual city of Chicago's system is among the best in the country. Is New York's better? Yes. But New York might be the only US city that can easily make that claim.
There is no comparison to be made between Chicago and San Francisco, or between Chicago Metro and SFBA.
This is the BART map: http://sf.metblogs.com/archives/images/2008/02/big_bart_map....
This is the El map: http://sites.williams.edu/thea228/files/2011/09/Chicago-El.j....
This is the New York subway map: http://images.nycsubway.org/maps/calcagno-2010-06-28c.gif
There is no contest.
Note also that the Chicago map is just a route map, and bears little relation to the geography of Chicago.
I have worked downtown for the last four years, and only drive to Montana.
Unclear to me what more than adequate might look like.
Also being a city of singles and young people, it creates it's own sort of gravity well and you get this magnetic alignment effect. You get married families in the south bay, leading to boring suburbia and the young in SF, leading to all of the fun things up there.
http://www.greatschools.org/find-a-school/moving/slideshows/...
Amusingly (considering how unlikely SF is to implement libertarian ideas), reason magazine kind of liked SF's approach
http://reason.com/archives/2006/04/01/the-agony-of-american-...
This all has come from conversations with people who grew up in the bay area, so the 50 minutes on muni might be different now from their childhood, but the top 3 school choice thing came from a coworker today talking about their friend's experience with their kids.
Your odds of getting something on your list depends of course in large part on the list. SF has a number of what we call "trophy" schools, as well as a number of highly in demand language immersion programs (Mandarin, Spanish, etc). If you list only those programs, your odds of getting skunked are high. Some people do this strategically, as people who went 0/7 on their list have a higher priority on wait lists than people who got one of their choices. They figure if they get one, great, if not, they have a priority position on the wait list. Other people play it a bit safer.
My approach was to avoid the trophy schools and focus on very good schools below the radar. In my opinion, there are plenty of very good schools in this range, but this is a matter of opinion. Here's the greatschools list.
http://www.greatschools.org/california/san-francisco/schools...
I did get one of the schools on my list. That's partly luck, and partly going in with realistic expectations (even my first choice was only an 8/10, I didn't go for the 9's or 10's). It is absolutely possible to create a balanced list with a good chance of an assignment and still get skunked, it's just less likely.
There's good and bad in the ranked choice system. San Franciscans are very split on it (a recent non-binding vote to make district residence the determining factor in school assignments, prop H, actually failed by a few hundred votes, as close to a 50-50 split as you'll ever see in an election).
I think the reason article summed up the benefits from a libertarian point of view pretty well, so I won't expand on it here.
So cities were planned around this. San Francisco itself even had a boom where it tried to utilize cars as much as possible. There were either freeways or plan to build more freeways cutting through the city to get to places. There was going to be a 2nd Bay Bridge to San Leandro, why? Because they wanted more traffic to get into the city and they didn't trust his new fangled thing called BART (the train system for those of you who don't know it).
I don't know what the situation was in England or other European countries when cars started taking off but I imagine there had to be something similar, maybe just not with the same momentum as the United States.
http://lostsf.wordpress.com/2010/11/07/the-elephant-obstruct...