Top colleges are incredibly competitive. Plenty of valedictorians and budding star athletes, artists, scientists, inventors are not admitted. To admit someone who's said racist remarks is to spit on the talented and hard-working "just missed it" applicants who presumably have the judgement not to hold (or at the very least say) such awful things.