Off the top of my head, you might get SF State students, tech bros, Chinatown senior citizens who have never left an 8-block radius and don't speak english, Marina moms, Mission District multi-gen families. I mean, come on.
Maybe if you were only picking from people working at tech cos, but even then my experience does not match yours.
So while OP may be wrong about a random sample of people in SF, they're probably correct about the people that they know in SF.
In a small town everyone shops at the same store, visits the same parks, works out at the same gym. There's only one library and a few restaurants, so there are fewer opportunities to self-select into smaller groups.
If OP can't find differing personalities in a place like SF, it's a skill issue, sorry to say.
That's the point, isn't it? You need to try, and have both the determination and "skill", to find variety when the environment gives you homogeneity.
I'm sure I could find an elementary school teacher, or an insurance salesperson to chit-chat with in SF or NY. I never tried, cause why would I? These aren't Pokemon to collect. It just so happens that my gym in the middle of Manhattan filters quite strongly for a certain income level, and thus age, but also profession, etc. In my current gym we all just go to the same class together.
Next time I'm in SF or NY I guess I can invest the time and effort to join a facebook group to meet school teachers, if that's what we mean by skill here.
From what I've seen from these large cities, the only reason people think that is that they remain in their small subset, which is large enough for them not to notice the rest.
If you're so inclined, sure, your small rural town is too small to have more than one community, and so there will be a little bit of social diversity. But if you live in a large city and are willing or need to go out of your in-group, the diversity is much larger.
There's a double crux here; Firstly, why would you? By sheer numbers I'm sure there's more school teachers, insurance salespeople and tattoo artists in NY than where I currently live. But while in NYC, why would I have gone out of my way to meet people with these professions specifically when there was plenty of people demanding my time in my circles already? Where I currently live, I just meet them at the gym.
Secondly, this applies to the other side too. While in SF I once people went to a party with a group of people that was very into photography. I never saw them again. Thinking about their perspective, why would they have hung out with me again? We lived far from each other (Oakland vs SF), they had own clique, and I'm not really into photography. The pull was weak both ways. I don't share much with the insurance salesperson I met at the gym, but it's the daily routine that's led to a meeting point, without any side having to "try", chatting, etc.
NYC has diversity by the numbers, but if you talk about an individual's perspective, there's very strong clustering leading to local homogeneity. SF isn't that big, so I'm not even sure if it has more diversity by the numbers compared to where I now live. But the filters leading someone to move to SF are strong, as is thus the pull to cluster with people that are very similar to you based on where you live, shop, workout, etc.
Now you have to be a lawyer or work in finance to hope to even get a modest sized apartment in NYC.
I've experienced the opposite too, having lived in both NYC and San Francisco: urban homogeneity, and even monocultures.
Yes, San Francisco is very diverse in the origin of its people, but the people who move there tend to fit certain molds, regardless of their cultural background. New York might attract more varied "types", but the act of moving to NYC still tends to select for a certain socio-economic level, a willingness to make certain sacrifices, people with life-goals and expectations, etc.
I surprisingly have found more diverse personalities and ideas in smaller places that are less selective (in price, profession, ideology, etc) to move to or to live in. Places where a software engineer might frequent the same gym as an insurance salesperson, an elementary-school teacher and a tattoo artist, even if (and perhaps because) they're all from there and didn't move in for a job.