Well the question is interesting when you look at the legal aspect instead of the 'I feel' aspect. If he deliberately refuses to play with some people based on a protected trait, it might be illegal. This seems like a much less clear cut case than e.g. having women and men basketball leagues. Your claim of 'freedom' doesn't (always) hold, you're not allowed (everywhere) to discriminate in all cases because of 'freedom'.