Even isolated prejudices can lead to systematic issues, so even if it was, it would be a problem.

For example, if key individuals in certain positions hold the prejudices, it can have a systematic effect, since they have a lot of influence on the design of the systems in place and can easily create an unfair distribution of other people in charge/not in charge.