Are you certain that you knew every black student at your school or just the popular ones? If there are few blacks it makes sense that the ones you'd notice would be tall and athletic as these are qualities that tend to engender popularity for school children. Did you conduct a formal or even semi-formal survey? I've heard similar suggestions to this before and they rarely conform to reality. This sounds more likely to be a case of conformation bias.

When you go to a school with very few black people, noticing one is already surprising enough that you won't miss many through confirmation bias.