During the civil rights movement, racist businesses and other associations hid behind this excuse. Some restaurants famously refused to serve people who were black, Italian, or Jewish.

But it is now pretty much settled law that people cannot weaponize their freedom of association to discriminate on these grounds. If you offer a service, you have to provide the service to the classes of people that you could be reasonably expected to serve.