To take an extreme example, the Muddy's cafes (near 16th and 24th) are one step away from dissolving into chaos. The people who work there seem to hate it.
RR has better coffee than average, and more foot traffic. I'm not sure why this hasn't been replicated. Maybe the beautiful people who seem to live there perform the same function as go-go dancers at clubs; they delude you into believing you're as hip as they are.
The best coffee experience in the Bay Area, as far as I know, is actually in a strip mall in Cupertino. Really!
Barefoot Coffee Roasters have many people on staff all the time, and they even hang out there in their off hours. It's got a very strong community feel and their baristas win a lot of awards. Much more laid back attitude compared to RR. I can't figure out why they haven't taken over the universe yet. Possibly they have higher staff costs, but choose to keep it like that rather than sacrifice to efficiency. That's just a guess though.
From personal experience (I've been in the market for coffee shop businesses of all types in Los Angeles for a couple years - from family owned tiny mom and pop stores to franchises), it's all about the location unless you have a lot of hype and if you have both even better. Anyone can start a coffee shop and stock their display cases with local/well known bakeries and coffee providers, but you need all kinds of customers (both the people who occupy a table with only a $1 cup of coffee all day long and the people who rush in and out, both something you can't get if you are badly-situated) and not just a little something extra. The latter is typically a way to get people to come to your store rather than another close by, or a little perk for your customers so they'll return later.
Plus, there's always Starbucks and Coffee Bean and Peet's to deal with...sigh.
Okay, they do have one big real estate advantage: they bought giant spaces with room for a roaster. But is there any other way a cafe could serve something special that still attracts the high-profit foot traffic that turns Ritual and Four Barrel into community centers? I'd like to hope so.
The long-term tourists, aka those in love with the city and who find themselves unable to leave it, enjoy the "bars i cafes" that exist all over. Many, many of them privately owned, not part of a chain.
A big difference between San Francisco and Barcelona is the following: people's houses and work places are all in the same place (which does not mean that the inhabitant's don't commute anyway). If anything, this overlap provides business throughout the entire day. This overlap provides "eyes on the street", that Jane Jacobs described, in her book "Death and Life of Great American Cities", as the source of safety and comfort (and thus business opportunities) in any city street, along with very small blocks so that walking around the local neighborhood is possible.
Perhaps in the big cities of the USA that match the Los Angeles sprawl model, like San Francisco does to some extent (as opposed to Manhattan & Brooklyn), only chain cafes may survive, riding on the low costs of mass production and distribution.
The Mission district is particularly local-oriented since there is a large youth and student population and a lot of professionals who either consult or work from home. Hence... cafes. Although this is more of a district that people live in during their 20s rather than their whole lives.
San Francisco's Valencia Street (with its immediate offshoots) supports hundreds of interesting businesses. Many of them are cafes. Only two of those cafes are any good: Ritual Roasters, and Four Barrel. Four Barrel was founded last year by a disgruntled co-founder of Ritual.
While the rest of Valencia's cafes are grimy, poorly lit, deathly quiet, or all of the above, RR and FB are sunny, friendly, and loud. That noise is all the foot traffic (and some David Bowie). The people sitting in Ritual for five hours on a laptop aren't paying the (staggeringly high) rent. It's the constant in-and-out of to-go cups. Mercifully, Four Barrel has no wifi and thus a more talkative, flowing crowd.
Ironically, because RR and FB have coffee that makes people come and go quickly, they can afford to make a pleasant space for people who stick around. Their solution to cafe economics was to make their own high-markup product.
Of course there are other solutions. Cafe du Soleil in the Lower Haight offers an extensive sandwich menu and gets its baked goods as part of a chain of boutique cafes and restaurants called the Bay Bread Group, which also sells bread to bigger clients like the Ritz Carlton. For all I know, Soleil is riding on the profits of the Bay Bread Group as an indulgent loss leader.
But I suppose there's a lesson there for any startup with a boutique aspect: Make your own supplies, and you might have a business plan to support your dream. In a way, wasn't that the model for Reddit and Justin.tv?