Correlation doesn't mean causation. If the cause is class bias, and you spend all your energy trying to make people more tolerant of other races, when actually they have no racist feelings whatsoever but have terrible class prejudices, then all your efforts are in vain. Trying to conflate the two things just because a correlation might exist is the problem OP is talking about.