Isn't the issue not refusing the group, but refusing work related to the group? Say there was a bakery that refused to make a NAACP cake, but which served people of any race from their normal selection of cakes? Even if one was to say that is wrong, it seems less wrong than fully denying any service to a person.