Freedom of association (or more specifically: the freedom to not associate) has limits; for example, freedom of association "doesn't matter" in the context of whether or not it's legal or moral to prohibit someone of a protected class from patronizing a business.