From a purely legal standpoint, the affirmative action issue is really about government funding and has little to do with limiting freedom of association. Schools that take federal funding can't violate the 14th Amendment's Equal Protection Clause. Private schools that wish to continue their affirmative action admission programs are free to do so provided they forgo government funding.