I'm not sure you understand how institutional and systemic racism works, but that kind of racism affects all people of color. That some individuals receive worse treatment than others in such systems doesn't mean that the lesser affected individuals aren't affected by that racism, nor does it mean that if you have other privileges (say, class) you are protected from the effects of those systems.

When it comes to implementing policies to address that, you look at who the systemic problems are affecting and you make choices to help address those effects. I'm sure some people with some arbitrary SSN are a person who is affected by institutional oppressions, but people are not targeted for inclusions in those systems based on SSN, so that's not a metric worth targeting when implementing policies (although if it can be demonstrated that there is a significant link between SSN and institutional racism, that's another story).

Things like racism (and other related -isms like sexism and to a lesser extent, ageism) are considered bad because they can be, and typically are, exercised against people based on largely uncontrollable aspects of their outward appearance. Everyone subconsciously creates associations between appearance, race, and social status throughout their entire lives, whether they realize it or not, and then makes judgments about new people they meet in light of those associations. Those judgments based on outward appearance are part of an initial impression then taint other subsequent judgments (and actions), such those as about a person's character or intelligence. Also, people learn that it's socially acceptable and generally expected to treat (say) a black person is with less respect than (say) a white person. And entrenched ideas about what people's social status ought to be cause a feedback loop that tends to impose these ideas on subsequent generations.

There are lots of other external properties that people are generally prejudiced for or against, such as weight/height/build, (dis)ability, posture, voice/speech properties, dress sense, and so on; but these (a) are considered to be more under an individual's control, (b) aren't inherited, and (c) historically haven't caused anywhere near as many social problems as racism in the US. No doubt people who are discriminated against based on their voice (say) don't like it, but it's not considered to be a systematic, self-reinforcing, widely-observed, entrenched social problem.

A hypothetical prejudice against "SSN % 104 == 7", where the property is not even outwardly observable (so can't genreally taint initial impressions), nor subject to this ongoing reinforcement, nor passed down through generations (neither the prejudiced property, nor the prejudice itself), is completely different from race, even moreso than the other examples.