Running a whites-only business overlaps but is different from being racist.
In this scenario, it would be weird to force your business to associate with whites-only businesses. But those are illegal, so you're not forced to do that. You're only forced to work with racists that don't run whites-only businesses.
My overall point here is that it does not make much sense to object to the freedom of association arguments being made in this thread on the grounds that similar arguments were used to support discrimination against Black Americans prior to the Civil Rights Act. The Civil Rights Act recognized racism as a unique evil that required special legal remedy; it did not recognize racists as requiring special legal protection. So, yes, freedom of association permits you to avoid associating with racists but does not (in all cases) permit you to avoid associating with people of a particular ethnicity.