The argument against affirmative actions was largely rooted in the concept that admission should be based on purely meritocratic considerations.

Legacy admissions are the opposite of meritocratic considerations, unless you believe that genetic inheritance is such an excellent predictor of cross-generational merit. Such a belief suffers from the same basic flaws as more clearly racist ideas.