Just looking at racial demographics, the level of diversity is about the same, it just skews differently. The % white and latino are about the same, but Chicago has as many black folks as San Francisco does asian folks, and vice versa. San Francisco's quite a bit more wealthy, though - median household income upward of $73k vs $47k. Perhaps that contributes to more of a perception of diversity for certain folks.

Chicago's hypercentralized layout may play a part there, too - the downtown area where tourists go is really just for tourists, and to some extent it's particularly for tourists making weekend trips from the suburbs. With the possible exception of Chinatown, all those great neighborhoods you're mentioning are pretty far off the beaten path for a visitor.