But that doesn't mean you can't live in Forest Hill, or the Excelsior, or Balboa Park, or West Portal. I find lots of people totally discount what I think are some of the nicer parts of SF to live in because they're not walking distance from the mission.
If I were a little more amenable to driving/riding, I could easily see myself living in a quieter neighborhood, given that I certainly value the ability to have a quiet(er) place to live and sleep soundly!
The K, L and M lines of the Muni Metro serve Forest Hill, West Portal. The L line goes to the SF Zoo but the K and M (along with the J) lines terminate at Balboa Park station. Personally, I prefer the Metro over BART due to its cleanliness and frequency of service.
I'm not sure about transit in Excelsior but I don't think it is that great.
Not trying to be pedantic but "willing to look slightly outside" is pretty ambiguous for a city like SF. The Tenderloin is slightly outside of Nob Hill, a very upscale and posh/safe neighborhood, whereas the Tenderloin greets you with used heroin needles on the street, homeless crackheads running around, and the noise of constant sirens.
I agree with you on affordability, though. Number one reason why I'm glad I'm back in the South Bay. Also, I'd rather enjoy the laid back nightlife of Los Gatos or Campbell over the Marina or Polk Street (20 year old sorority girls puking all over the place) any day.