Having gotten my ass kicked intellectually repeatedly by black and hispanic students: no, those backgrounds are not a liability.

The two are not mutually exclusive. I would argue that a very significant part of the problem that Blacks and Hispanics face (besides simple discrimination on the part of people in positions of power -- which is very real, as studies of responses to applications in various context from people with stereotypically Black or Hispanic names vs. otherwise identical applications without that feature) is attitudes toward societal institutions and mainstream culture in their respective subcultures that are a direct product of problems in the broader society (including the schooling system), both currently and over the past several centuries (the longer time especially for black subculture) including, over much of that time, overt de jure subjugation and exclusion (which while it didn't directly impact young people today, is recent enough that lots of them have living relatives who did directly experience it) followed by covert (but still present, see earlier comment about applications) discrimination and neglect.

Arguing over whether A or B is the problem is pointless when A causes B which then contributes back to A.