That's... strange. SF's black population has definitely been on the decline (but they're certainly here, all over!), but you must have been actively hiding if your only exposure to brown people was people "kept back in the kitchen".

I'm not going to claim that SF is some amazingly perfect bastion of racial diversity, but if you're mainly only seeing and interacting with white and Asian people here, that speaks more to the circles you spend your time in than the city's diversity.